Alexey Maykov

bulk insert access denied problem

2009-02-16T13:24:00.001-08:00

You may get this error when trying to use the bulk insert operator in SQL server 2005:

Cannot bulk load because the file "\\maykov100\public\blogcatalog.txt" could not be opened. Operating system error code 5(Access is denied.).

As it turns out, SQL server tries to use your credential to access the file to bulk import (impersonation). In a situation, where you access SQL server remotely (ie from a SQL Management Studio running on your machine), your credentials can't be retranslated. Possible ways to solve this problem:

Use BCP from your local machine
Run the SQL managements studio directly on a SQL machine
Modify your company's active directory to allow impresonalizati0n

Visual Studio, Remote Debugging, DOM problems, etc

2009-02-04T18:13:00.001-08:00

I think, this is familiar to any Windows developer: you start a remote debug monitor, point your VS to it and all you get is some network error message box. Today, I finally solved this mystery. Make sure that you can access the machine running Visual Studio from the machine where the program is running by its DNS name. In my case, I just got assigned a new IP address from DHCP and the remote machine had a wrong IP for my machine. Once I fixed this, remote debugging started working again.

C# HttpListener: HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request

2009-02-04T18:09:00.001-08:00

In this blog, I'm going to start focusing more on small details of C# programming. With this said:

HttpListener is a great C# 2.0 class. It allows you to create web-servers without having to have an IIS server on your machine. This became possible since a while ago, IIS was divided into two parts: HTTP.SYS and proper IIS. HTTP.SYS is supplied with any machine running XP SP2 or Server 2003. HttpListener allows .net programs to use HTTP.sys

My program creates HTTPListener, calls Start and BeginGetContext methods. Then, the thread terminates. Whenever an HTTP request is received a delegate is called on a thread pool thread which call EndGetContext and generates HTTP request. It worked fine on Vista, but started throwing exceptions on Server 2003. After I made sure that the thread calling HTTPLIstener.Start didn't terminate, the exception went away.

I hope, this will be useful to some pure soul trying to debug.

Snow

2008-12-26T15:20:00.001-08:00

Seattle area is not a snowy place. When a snow fells several times per year, usually it doesn't stick for longer than one or two days. This year, we're having an anomalously long period of snow. In the last week, we went through all phases of a typical Russian winter. It started with a heavy December snowfall on Thursday, then there was a January low temperature and a sunshine on Friday and Saturday. A February blizzard followed. Now, we're going through the early spring snow melting. We and other people in my neighborhood got a full advantage of snow activities. We went sledding, cross country skiing, built lots of snowmen. I even saw downhill skiers and snowboarders. Traffic, of course, was the mess. The lack of snowplows in Seattle manifested itself in a fact that they closed the steep road near my neighborhood. Some people kept using it since it was the only way to get to their house. Instead of plowing it, they put a policeman giving out warnings and tickets for driving on that road.

Here are some photos which I took:

Sledding hill:

US Mail truck making it through the snow:

The UPS truck got stuck and was abandoned for the night.

An Igloo house which we built on our lawn.

Magnificent!

2008-12-01T21:47:00.001-08:00

VIdeo projection Monumentale 2009
Uploaded by uruk

Very nice LiveJournal visualization

2008-11-10T20:44:00.001-08:00

http://aqua.livejournal.ru/

BTW, all these д nonsense on the background is a porno spam in Russian language.

Politics

2008-11-03T23:02:00.001-08:00

As we are getting closer to the unwinding of the current political conundrum, here is a video which may help you to make your decision. Here we go: GOP maverick

PPTPlex

2008-10-14T22:09:00.001-07:00

A very cool technology from Microsoft Office Lbs. Allows presentation in a style close to the visualistion on my previous post

<a href="http://video.msn.com/?playlist=videoByUuids:uuids:f362631f-c86c-4547-a544-9b8eda9975e3&showPlaylist=true&from=shared" target="_new" title="An Overview of pptPlex">Video: An Overview of pptPlex</a>

Political Streams

2008-10-09T10:11:00.001-07:00

Check out the product which we just released: http://socialstreams.livelabs.com/politics/ . From FAQ: Political Streams is an application which mines social media content in real time for political discussion. It surfaces the news articles and documents that are being discussed as well as the people and places that appear in those articles.

A number of great people worked on it. Matthew Hurst and Justin Rudd are among them and have blogs.

Why we drink

2008-09-19T15:02:00.001-07:00

Amazingly cool visualization:

Here is a link to the author's site with even more cool stuff: http://www.clemenskogler.net

My Readership

2008-09-15T20:55:00.001-07:00

Interestingly, the search engine query which brings the most visitors to my site is "japan[ese] pornsite". (See the related post here). May be, I should change the topic of my blog to something more mundane and timeless...

Relational Databases are Considered Harmful for IR Systems

2008-09-08T21:26:00.001-07:00

When building a new system, most developers would throw in a relational database without thinking too hard bout it. After all, RDBMS are designed to deal with data. However, they may be not a good fit for some kind of systems. I'm talking about systems which deal with terabytes of (text) data. I'm referring to such systems as IR (Informational Retrieval) systems, but I'm not limiting this discussion only to IR. Of course, there are tons of systems where SQL server is still useful. Let's talk about concrete examples of both. We will use a web search engine as an example of an IR system. We will use a banking software (ie the software which bank runs) as an example of a traditional system. I'm going to briefly compare two in several dimensions. Later on, I will go into details on each.

	WEB Search Engine	Bank System
Number of simultaneous writers	one	thousands
Number of simultaneous readers	thousands	millions
Cost of a Data Loss (per GB of data)	low	extremely high
Data Consistency	not required	required
Data Volume	Terabytes	Gigabytes
Transactions	Few long-running	Multiple short-running

In a bank system, thousands of users make deposits, transfers and so on. At the same time, in a web search engine there is only one (or a handful) of processes which update the index.

A Web search engine has an extremely high number of simultaneous users. A banking system has a moderate number.

A loss of any data my be catastrophic for bank and may potentially disrupt the business. A search engine can afford loosing some of its index. At the very worst, it will recrawl affected pages.

It is important that data is consistent across all bank databases. A canonical transactional processing example involves a money transfer from one account to another. Transactions like this should be either completed or not. Money should be withdrawn from one account and deposited into another. Cases when the money is only withdrawn or only deposited are not allowed. For the web search engine, on contrary, it is quite alright when the index doesn't reflect the latest changes on the web. The update may be lagging several days. Different copies of the index may have different versions of the same page and this is fine.

In a banks there are lots of simultaneous short running transactions such as money transfers, withdrawals, deposits. A web search engine does transactions on big batches of data. I must say, this item is more of an implementation detail.

Finally, the volume of data in a search engine is much higher.

Clearly, a RDBMS supports banking scenario well. A wealth of thought has been put into allowing multiple simultaneous transactions.Under no circumstances, these transactions will put a database into inconsistent state. A typical RDBMS allows setups with high protection against data loss. These properties of RDBMS are not needed for a search engine. The also come with a high price both in terms of money and performance. Paying this price when not needed is hardly justified. For instance, typical SQL server configuration is several times more expensive then a typical commodity server. SQL server overhead in terms of a disk storage may be an order of a magnitude (SQL overhead, RAID mirroring, dump space, log space, etc..).

What can be used as an alternative to RDBMS for building large IR systems. I'm not sure as I'm only starting in this field. Feel free to suggest. One alternative would be to build your own custom data structures. I'm going to demonstrate this on several examples in coming posts. Right now, I'm thinking about describing an inverted index and a web crawler. Feel free to suggest your own.

Google's street view flashmob

2008-08-06T11:48:00.001-07:00

Somehow, Google employees found out about when their street was going to be pictured for the street view. So, they came out and started to act out: http://maps.google.com/?ie=UTF8&layer=c&cbll=37.420887,-122.083965&panoid=0JwQNpGw9ctY-fZ7BA0_dA&cbp=1,6.385842491873461,,1,9.789184299225807&ll=37.425781,-122.084069&spn=0.009457,0.014248&z=16&source=embed

Parallel Crawlers

2008-07-15T00:11:00.001-07:00

Junghoo Cho and Hector Garcia-Molina "Parallel Crawlers." In Proceedings of the 11th World Wide Web conference (WWW11), Honolulu, Hawaii, May 2002.

The paper mainly concerns different partitioning/communication techniques in building parallel crawlers. The space of all URLs to crawl may be partitioned by: URL hash, host hash, etc. Crawlers running on multiple machines may send links to each other, may crawl all links instead of sending them or may ignore links which are outside of their partition. The paper studies pros and cons of each approach.

I think, that the paper is lacking on discussing performance, fault tolerance. But it is nice to have papers like this as they help generate/validate ideas.

Japan: 780,000 porn site hits earn demotion

2008-06-26T15:02:00.001-07:00

A Japanese civil servant was demoted for logging more than 780,000 hits on pornographic Web sites on his office computer over nine months, an official said Friday... the man logged 170,000 hits on porn sites in July alone.

http://www.msnbc.msn.com/id/24426097/

This totals to about 14 hits every minute! This man was either running a porn crawler or some spy-ware was hitting these sites.

Commute: Gas Prices

2008-06-24T11:45:00.001-07:00

Met a suburban driver at a gas station today. It takes $180 to fill up her Suburban. I noticed, she didn't fill it up this time. Another tidbid: San-Diegans drive to Mexico to buy gas.

With a high price of housing, some people drive till they qualify. Let's say, somebody bought a house in Snoqualmie ridge and commutes to the Microsoft main campus every day. It is 50 miles per day or 1100 miles per month. GMC Yukon 1500 AWD 8 cyl, 6 L, Automatic 4-spd, Premium consumes 16 miles per gallon. At a fuel price of $5 per gallon (got to think ahead), it'll be $343 per month or $4125 a year for fuel. This will buy at most $80000 more of a hous (at APR of 5% and no interest). The gas prices have at least double to stop the urban sprawl.

Interestingly, you pay not only the gas but also your time for a long commute. Here is another perspective: http://noisetank.com/hugeasscity/2008/04/14/busting-the-drive-till-you-qualify-myth/

Trying out Public Transportation

2008-06-23T23:34:00.001-07:00

I was trying a public transportation in one form or another in the last week. Mostly I was using Microsoft provided shuttle buses. These buses are designed for carrying people from one building to another with the exception of a connector. The connector is a network of luxury coaches for getting to and from work. Microsoft also provides a free pass for all kinds of public transportation. Needless to say, public transit is scarce. There is a bus which stops at a walking distance from my home at Redmond Educational Hill and goes directly to the downtown Bellevue where I work. The very first outgoing bus of a day makes a stop at 5am and the last one stops at 8am. Clearly, this doesn't work for me. The remaining option is to take a connector and then a shuttle. This takes an hour, while driving takes 20 minutes. I really miss subways in Moscow. Tomorrow I'm going to try biking to work. Another thing to try would be to get one of the vanpool vans. King county provides a van for people who would like to carpool. The van includes maintenance and gas. And the cost is covered my the Microsoft bus pass. I have at least 4 other Microsoft friends living nearby.

Egg-throwing Student

2008-05-20T13:33:00.000-07:00

http://www.youtube.com/watch?v=mtBQ4UCXQeo

Threw some eggs at Steve Ballmer.

Trading Speed for Memory and for Latency

2008-05-19T22:38:00.001-07:00

It is a widely known that you can trade memory for speed in some circumstances. For instance, instead of doing lengthy computations each time, pre-compute, store results in a memory and lookup the result at a run time. Another example is caching: RAM or disk. A less known fact is that sometimes you can significantly speed things up by the price of allowing more latency. For instance, instead of sending data over the wire, burn them on DVDs and ship them.

Here is an interesting technique which is often get used in crawlers and was described in the IRLBot paper: IRLbot- Scaling to 6 Billion Pages and Beyond. The problem is that for each new URL, you need to determine among other things, if you have already crawled it. It is logical that you will need to have a database of seen URLs. The number of lookups and/or inserts per second will determine a potential crawler bottleneck. A conventional technique of a DB lookup per URL will soon hit a wall of scale as a number of Seen URLs reaches into billions. Instead, lookup request can be batched and than executed all at once by doing merge. Merge is a much for complex operation than lookup. However, lots of requests may be executed in one merge operation while each request will require a separate lookup.

WWW Opening ceremony

2008-04-22T19:03:00.001-07:00

What was the reason behind one of the dancers being a male and all others female? I hope, not an attempt to play by political correctness rules.

Kai Fu Lee Keynote

2008-04-22T19:01:00.001-07:00

Mainly, Google propoganda, reiterating known things.

Cloud computing, everything moves to the cloud, same old story. He showed a sample when Google's machine translation from Chinese into English is better than it's competitor. This shouldn't belong to a keynote speech.

One thing which I didn't know, Google can deploy a new data center in 3 days. Their boxes (i.e. machines) do not have boxes :). Google came up with a way to stuff motherboards directly into racks. It is unclear where hard drives are located.

@Beijing for WWW '08

2008-04-20T09:31:00.001-07:00

The livejournal is still denied for access from China. So much for "One World, One Web", the theme of the conference.

Great tutorial day tomorrow

PhD in Computer Science

2008-04-16T18:09:00.000-07:00

A great essay on PhD in CS:

http://www-2.cs.cmu.edu/~harchol/gradschooltalk.pdf

TREC Blog Track. Nominal? fee.

2008-04-15T00:00:00.001-07:00

Price £400 = $787

Total Size: 135 gB on 3.5" hard drive=$70 for hard drive

Number of HTTP requests: 324,880 homepages + 3,215,171 permalinks + 100,649 feeds * 11 weeks = 4,647,190

This number of requests in 11 weeks results from about 41 requests per minute or less than one request per second. I am averaging 20-50 requests per second on on my office box. My home Comcast cable service allows only 2 simultaneous requests and a page load every 2 seconds is not a wild assumption. The service cost for 11 weeks is about $100. So, you cane save around $600 minus unknown amount of dev and ops cost if you choose to run your own crawler. However, you won't be able to submit a paper to TREC.

ICWSM Papers

2008-04-14T23:26:00.001-07:00

Here are some highlights.

Lots of papers on LSA-LDA. I am fascinating by the elegance and non-supervision(?) of this technique.

•Link-PLSA-LDA: A new unsupervised model for topics and influence of blogs

•The Psychology of Word Use in Depression Forums in English and in Spanish

TREC

This is a conference where all papers are about solving a fixed set of problems on a given set of data. So, everybody understands 100% of all papers and fully engaged.

•Document Representation and Query Expansion Models for Blog Recommendation basically, the winner of Blog'07

•On TREC Blog Track. The dataset is available for a nominal fee of 400 pounds.

With over 200 world languages and the most widely used language being oddly the simplest one, I thought that good language coverage can be achieved only by teams of zillion people. However, there are some tricks with machine translation:

•International Sentiment Analysis for News and Blogs

Keeping on

2008-04-12T12:07:00.001-07:00

I must say that keeping the blog going is much harder than starting one. You start all excited and jazzed up and then do not have a motivation to keep going. I saw lots of blogs with only a few entries. It would be an interesting study to see what the distribution of the total number of posts look like and check if there are any correlations with other parameters.

ICWSM '08, Day 0

2008-03-30T19:22:00.000-07:00

Today I attended a Day 0 of ICWSM '08. ICWSM is the conference organized by a group of researches in the Social Media area along with Matt Hurst.

Both tutorials were quite interesting. Jan Wiebe talked about subjectivity and sentiment analysis. According to some conference attendees, sentiment is a hot topic right now. Lots of business people would like to use the results of a sentiment mining. However, they're sceptical about how reliable these results are. Jan provided lots of details about different type of sentiment and objectivity-related annotations which exists for a data. The analysis part though, wasn't quite evolved.

Mary McGlohon talked about different ways to study big graphs. Examples of big graphs are: web-links graph, frend graphs on social networks, blog links. There are interesting properties of measures of such graphs, such as diameter, in/out-degree. Since graphs are also matrices, diagonalization tricks such as SVD can be applied, which lead to interesting results. Interestingly, the evolution of graphs in time can be studied using tensor analysis. Turns out, tensor is a three-dimensional matrix. I really should've paid more attention to my Math and Physics classes instead of hacking C++ code! I highly recommend checking out Mary's slides at http://icwsm.org/2008/tutorials.shtml

Tomorrow will be a great day which will start with a talk by a LiveJournal founder Brad Fitzpatrick. Stay tuned for more updates!

Introduction

2008-03-29T23:08:00.000-07:00

Hello!

My name is Alexey Maykov. I live and work in Redmond, WA. I've been with Microsoft for 7 years. Last two years I spent in Live Labs.

I'm working on data mining Social Media. My immediate coworkers include Matt Hurst, Dragos Manolescu and Justin Rudd. The rest of the team doesn't have blogs yet, so I won't reveal them yet.