Well only just… I’ve got some results from my attempt to divide up the feral druid population into cats and bears. We started from the fact that there is no “form” tag in the armoury XML – no direct way to count the thing we want to count. The only way to get an insight into this is to find a proxy for each of the forms – something that is in the data which can be used to separate the sheep from the goats, if you’ll pardon the mixed metaphor.

Talents seem to be the obvious choice, so long as there is one talent that bears will take and cats not and another talent that is vice versa. Glyphs are the other possibility. Whatever we choose just has to be i) something that players are highly likely to take  and ii) something that is orthogonal; something that definitely points in one direction for bears and another for cats.

But the basic problem is that there are a lot of um… how to put this politely… there are a lot of left-of-centre specs out there. Talents and glyphs are both less orthogonal than I was hoping for – many specs look a bit bearish and a bit cattish at the same time. And there is a big group that takes none of the talents or glyphs that we want to use.

That’s why I decided not to make the queries very complex – adding more talents or glyphs into the selection criteria  just increases the number of toons that fall into the grey area. Also I’ve counted specs and not toons since the original question was related to the number of druids specced for tanking.

Thanks to the commenters who made suggestions on possible talents and glyphs that might fit these criteria. I’ve run two queries against the data.

The first query counts feral druids who have Natural Reaction versus those who have Predatory Instincts. A druid with some points in Natural Reaction and none at all in Predatory Instincts might be a bear; t’other way round for cats. Those with points in neither are marked as “unknown”; those with some points in both are the “could be either” group.

The second query counts druids who have a Glyph of Maul versus those who have either a Glyph of Shred and/or a Glyph of Rip. Equipping Maul but not Shred or Rip indicates bear; Shred or Rip but no Maul indicates cat. Again we have groups with a mix of these glyphs, and, unfortunately, a huge group with none of them.

Anyway this is what we’ve got:

(Patch 3.2.2 data; sample size 16327 level 80 feral druids with 28970 specs).

Talent-based spec count:

• Bear: 30%
• Cat : 33%
• Could be either: 5%
• Unknown: 31%

Glyph-based spec count:

• Bear: 18%
• Cat: 9%
• Could be either: 15%
• Unknown: 58%

Frankly I’m still not sure how valid these numbers are, but I hope they provide a bit of insight. The talent-based count may at least provide a low-water-mark indication of the number of bearish specs in there.

About a month ago, Dan Rockwell and I finished writing an article for interactions magazine about Casual Data, the term we’ve used to describe rich data propagated or mined via some form of social media. The piece defines Casual Data, talks briefly about why it’s becoming so prevalent, and then proceeds to identify current ways it’s being used and what that means to the fields of design and research. It will be out in a spring 2010 issue of interactions, however, here’s a sneak peek at some of the data nugget goodness:

The problem with too much data
While there are a number of firms analyzing the surface value of casual data, there is a need dig deeper to understand context and higher-level implications. The more connected we become, the more connected our data becomes, and the more we need a structured approach for making sense of it.

Companies having loads of customer data available is not news, however this casual data is not quantitative in nature (demographics, pattern-focused). The emotional meaning behind casual data should not be analyzed statistically, and the methods used to gain this data are as important to understand as the data itself. If customer voice is only harvested through an existing medium (e.g. submitting a query for iPhone-related tweets) the results you get will be brief and will tend to either be of intense glee: “new iPhone copy/paste function, thank GOD” or intense distaste: “Apple sucks!” – leaving little room for understanding context of use, while still providing good touch-points for product improvement. There is the potential of casual data being more dangerous than helpful if not properly understood.

I det direktivudkast, der blev vedtaget på rådsmødet d. 26. oktober og omtalt i blogindlægget d. 3. november, præciseres borgerens privacy ved at fastslå, at navne, emailadresser og bankoplysninger, data om alle telefonsamtaler og internetsessioner, skal opbevares sikkert med henblik på at undgå uheld eller at disse data med vilje falder i de forkertes hænder. Det fremgår også, at brugeren skal have klar og fyldestgørende besked om hvorledes der er forholdt med hans data og at han skal give sit samtykke til at hans data gemmes og at andre kan få adgang til disse data.

Dette er på engelsk formuleret således i udkastet: ” Member States shall ensure that the storing of information, or the gaining of access to information already stored, in the terminal equipment of a subscriber or user is only allowed on condition that the subscriber or user concerned has given his or her consent, having been provided with clear and comprehensive information, in accordance with Directive 95/46/EC…”

Denne revision af EU´s databeskyttelsesdirektiv kan vise sig at få en endog meget stor negativ indvirkning på online-annoncering med kravet om, at annoncører på forhånd skal indhente brugerens samtykke før de kan placere de såkaldte cookies på deres servere med det formål at effektivisere reklamepraksis ved at gennemføre en målrettet kommunikation og markedsføring ( personligt identificerbare forbrugerdata).

En cookie er betegnelsen for en tekst-fil, der for en bestemt tidsperiode er gemt på en klient på vegne af en server. Servere gør normalt brug af cookies til at gemme brugeridentifikation, brugeradfærd og indkøbsvaner. Cookie´en bliver sendt tilbage til serveren ved senere forespørgsler fra klienten.

EU-Kommissionen har gentagne gange udtrykt bekymring over indsamling af forbrugerdata til brug i online-annoncering og samtidig slået fast, at den ikke vil tøve med at gribe ind, såfremt branchen ikke selv aftaler et regelsæt, jfr. eksempelvis EU-lovgivning om online-annoncering på vej. Med det foreslåede direktivudkast synes EU-Kommissionen ikke mere at afvente en tilfredsstillende branchekodeks. Forholdet er det, at flere branche-organisationer har udarbejdet adfærdskodeks for online-annoncering, men de giver kun forbrugeren en opt-out og ikke en opt-in mulighed (forbrugersamtykke). Principperne om gennemsigtighed og forbrugervalg synes derfor ikke at være opfyldt.

Meget kunne tyde på, at direktivukastet som en del af “telepakken” vil blive endelig vedtaget af EU-Rådet og EU-Parlamentet inden årets udgang med den konsekvens, at ISP´ere, som f.eks. Google og Microsoft samt en en lang række annoncenetværk vil blive tvunget til at indhente brugerens samtykke, før indsamling af data med henblik på brugerens interaktion. Cookies vil kun være tilladt uden direkte brugerens samtykke, såfremt de er “strengt nødvendige” for at yde en service brugeren “udtrykkeligt” har bedt om, som f.eks. at gemme indkøb via hjemmesider for nethandel.

Det er i skrivende stund imidlertid uklart, hvorledes de enkelte EU-lande, herunder Danmark, vælger at implementere direktivet i konkret lovgivning, men det synes at være udelukket, at komme uden om kravet om forbrugerens samtykke til oprettelse af cookies. Det vil uden tvivl gøre det meget vanskeligt at bruge hjemmesider og man kan stille sig selv det spørgsmål, om udbydere vil risikere at se bort fra loven i det håb at loven ikke vil kunne håndhæves i praksis ?

Branchen er sat under pres, men mon ikke den under alle omstændigheder vil kunne nå at udarbejde en kodeks, som i det væsentligste opfylder direktivudkastet og at man dermed undgår en lovregulering, som kan vise sig at blive mere rigoristisk og mindre adræt og alligevel ikke effektiv nok.

Pada Bab 2 ini akan dibahas salah satu langkah penting dari data mininng yaitu warehousing. Bahsannya meliputi, konsep-konsep, berbagai istilah, karakteristik, manfaat, tujuan, tugas-tugas data warehouseing.

Download pdf:

Bab 2 Data Mining  – Konsep Warehousing

There’s a post over at wow.com that has set me a bit of a challenge. The post is about bear tanks, but makes the valid point that we don’t have any clear data on the popularity of the various druid forms. There’s a pretty simple reason for that – the armoury data doesn’t provide any direct way of getting such a count.

Still, we don’t let little obstacles like that get in our way. What we need are some data items that can be used as proxies for what we want to count. Unfortunately I’m far from being a druid expert, so I’m looking for suggestions on what items to use.

What we basically want is a talent, or a glyph (or maybe a gem) that bears will want to equip and cats not. And then something that’s vice versa – something that cats will have and bears not. One talent or glyph, or several… whatever makes the most sense. All suggestions on this are most welcome.

(Thanks to the commenters who have already made suggestions on other threads; I’ll be taking those comments on board.)

If I can get suggestions for both talents and glyphs then I can run more that one query and see how well the numbers match up.

I’d imagine that, with dual specs, players who liked both forms would have a spec for each. In any case, going from specs to forms and getting a count against the total druid population should tell us something interesting.

Google and Microsoft eye eWorld domination
22 October 2009

Well, if you didn’t think these Google and Microsoft were trying to know everything about you… they just bought your microblog.  Twitter has just sold the rights to users’ tweets to the search giants for an undisclosed amount.

I’m not sure how I feel about Microsoft, but Google…  man, you took photos of my home, you read through my e-mails, my searches, now you’re in my microblog?  Stalker.

I mean, just when you thought the world couldn’t know anymore about you… Huzzuh for datamining!!!  I guess this is just a really harsh warning to everyone out there that nothing is free.  Get a free blogging account, sell your information.  Get a free email account, sell your information.  Get a free colorectal exam–wait, maybe that one doesn’t apply.  Still, I know everyone’s talking about the “information age” we’re in, but I’d like some privacy.  Possessions aside, the only things we really own are personal privacy and personal relations, right?

Original Story:

http://www.cbc.ca/technology/story/2009/10/22/tehc-internet-twitter-google-microsoft.html

]]> http://ubisurv.wordpress.com/2009/10/20/cia-buys-into-web-2-0-monitoring-firm/ Tue, 20 Oct 2009 14:28:38 +0000 David http://ubisurv.wordpress.com/2009/10/20/cia-buys-into-web-2-0-monitoring-firm/

Wired online has a report that the US Central Intelligence Agency has bought a significant stake in a market research firm called Visible Technologies that specializes in monitoring new social media such as blogs, mirco-blogs, forums, customer feedback sites and social networking sites (although not closed sites like Facebook – or at least that’s what they claim).  This is interesting but it isn’t surprising – most of what intelligence agencies has always been sifting through the masses of openly available information out there – what is now called open-source intelligence – but the fact is that people are putting more of themselves out their than ever before, and material that you would never have expected to be of interest to either commercial or state organisations is now there to be mined for useful data.

(thanks, once again to Aaron Martin for this).

Formado em Ciência Política pela Universidade de Michigan e professor convidado das universidades de Chicago, Loyola e Harvard, Peter Giangreco é um dos maiores especialistas em mala direta dos Estados Unidos e foi o responsável pela estratégia de marketing direto e microtargeting da campanha presidencial de Barack Obama. Giangreco é socio do escritório The Strategy Group e possui mais de 20 anos de experiência atuando nas campanhas de Bill Clinton e Al Gore, além de ter assessorado o senador e ex-pré-candidato democrata John Edwards.

Saiba como foi a participação de Giangreco durante o 1º Seminário de Estratégia de Comunicação e Marketing da George Washington University:

• O mapa do eleitor americano, segundo os democratas
• “Brasileiros devem investir no porta em porta”, diz estrategista

Assume for a minute that forecasts of a cloudy web are true. Enterprises enter a refactoring phase which complements phones replacing PCs amid meetups instead of meetings. More extreme cases result in, not just paperless office, but officeless professions, education and government. The schizophrenic screen that has spent half the time on work and the other on web2.0 then solidifies seamlessly. In the future tense, it is likely that there are applications where the cloud is useful at an acceptable threshold of trust.

Data volume being like a tornado that sweeps over engines of search, classification and computation, a data mining booster may help find direction again, and test data analysis may establish course correction. Tools have found their way onto PCs and networks in the QA, agile and open-source cases. Components can be customized to run off of portable media such as flash drive or CD, USB or P2P, browser plugins, and phone apps. The next exercise is to establish stable, sustainable and scalable eQuality. For example, a mashup of templates from TIS and cloud would vector versions of testability metrics, virtual toolsets, testing-as-a-service, cloud under test, and analysis on demand. Depending upon the cultural or commercial context, they may tend more to either social or automated implementations.

This potentially adds new mosaics to dashboards that are now limited to notes, feeds, reviews, wikis, communications, activities, spreadsheets, and media. Cumulative captains of industry can derive further case studies, cluster BOMs and business plans. Test teams may widen their range and discover unmet needs and niches as tech has a tendency to do. Exceptions to the cloud pivot toward continuing efforts at clearing constraints.

Thanks to reader Armagon who asked for a consolidated report on race distribution. This subject is covered by a variety of other sites, but there is some doubt about whether those sites are maintaining a representative sample.

It’s easy enough to produce a couple of simple tables that give us the information we need. (The data is from the patch 3.2 scan.)

That seems to be a reasonable match to the Warcraft Realms data. There will always be some degree of sampling error in this work so everybody’s numbers have to be treated with a bit of caution.

Tables like that always make me shake my head a bit but… the fantasy RPG where everybody wants to roleplay the cute kid next door… Hey, trolls are people too you know!

Anyway, if we want the distribution of race and class then we get this. (Percentages here are based on the total population so the popularity column adds up to 100%.)

I’ll add this to my set of reports over at my Google site, so that it stays updated with each new scan. But it does seem that the other sites that cover population are doing a reasonable job of reporting what is really going on.

The US House of Representatives will finally get to debate whether online advertising which tracks the browsing habits of users is a violation of privacy and needs to be controlled. A bill introduced by Rep. Rick Boucher of Virginia will be propsing an opt-out regime that gives users information about the uses to which their data will be put, and allows them to refuse to be enroled. At present many such services work entirely unannounced, placing cookies on users’ hard drives and using other tracking and datamining techniques, and without any way in which a user can say ‘no’. Of course, we have yet to see the results of the inveitable industry scare-stories and hard-lobbying on the what will be proposed, let alone pased. But the proposal itself is particularly significant because so far the US has so far always bowed to business interests on online privacy and data protection, and if this bill is pased, it is a sign that what EFF-founder, Howard Rhiengold, long ago called the ‘electronic frontier’ might start to acquire a little more law and order in favour of ordinary people.

Not having made use of PLINQ in an actual product yet, I decided to have a play with how it works, and to try and obtain my own small metrics on it’s performance benefits. PLINQ is part of a larger push from the .NET teams at Microsoft to get concurrent/parallel processing out of the box in your C# and VB.NET code. As for performance analysis there are already some great posts out there, not just from the Parallel Team at MS but also from great breakdowns with nice charts such as this.

Right off the bat, I’d like to stress that adding a .AsParallel() to your code won’t magically speed it up. Knowing this I still had unrealistic expectations when I began creating a demo to specifically show performance improvements. Often enough the level of processing I was performing (even on larger sets of data), did not benefit from being made concurrent across 2 cores. A level of variation in my results, leads me to believe part of the issue is also the ability to obtain enough resources to make effective use of 2+ cores. For example running out of the 4GB ram I have available, interference from other processes on the machine (Firefox, TweetDeck, virus scanner).

In my attempts at re-creating the “Baby Names” demo Scott Hanselman previewed at the 2009 NDC Conference in his great presentation: “Whirlwind Tour of .NET 4“. I first got a hold of the preview code samples back from 2008 for PLINQ that were part of the Parallel Extenstions CTP.

I then went on to from scratch create my own simple PLINQ – Windows Presentation Foundation (WPF) application.

I chose WPF to test a small feature I hadn’t made use of yet only because I happened to stumble upon it on that day; Routed Events see this StackOverflow question.

Once I completed my take on a LINQ processing demo based on 2 minutes of video showing the operation of ‘Baby Names’, I discovered (by accident*) the Visual Studio 2010 and .NET Framework 4 Training Kit – May Preview, which contains the demo code for what I was trying to re-create.

*The accident in which I discovered the Training Kit, was I actually performed a google image search on the term ‘PLINQ’ to see what came up for ideas for a graphic to add to this post. The 11th image (centre screen) was the baby name graph displayed in theWhirlwind Tour of .NET 4 presentation. The post that had the image was from Bruno Terkaly, the post was about the tool kit, great!

VS 2010 Training Kit May Preview

None the less, my not-as polished demo application, makes use of the StackOverflow creative commons data dump (actually the Sep 09 drop).

Some background: I grabbed the StackOverflow data dump via the LegalTorrents link, then I followed this great post from Brent Ozar, where he supplies code for 5 stored procedures to create a table schema and import the XML data into SQL Server. It was as simple as running them, and then writing 5 exec statements and waiting the ~1 hour to load the data (resulting for me in a 2.5 gig DB).

The way I structured a lengthy processing task that can benefit from parallel processing, is by making use of the Posts data (questions and answers), in particular questions with an accepted answer. I make an attempt through a repetitive simple string comparison process to determine how valid the tags on the question are, by scanning the question text for the tags, and counting frequency. Then timing the processing of sequential operation vs the parallel operation as I pipe varying levels of data into the function.

First I extract the data into memory from SQL Server (using LINQ to SQL Entities). Just a note on the specifics of the SO Data Dump structure; ‘Score’ is a nullable int so just to keep the data set down in volume I select posts that have some score and greater than a selected input (usually 10+ at least 1 person liked it), same with a reasonable amount of views (on average 200+).

private IEnumerable<Post> GetPosts(int score, int views)
{
   var posts = from p in db.Posts
           where (p.Score ?? 0) > score
           && p.ViewCount > views
           select p;

   return posts.ToList();
}

The next step was to create a function that would take some time to process, and of course potentially benefit from being run in parallel. Each post and it’s tags are operated in isolation, so this is clearly prime for separation over multiple cores. Sadly my development laptop only has 2 cores.

private bool IsDescriptive(Post p)
{
   //lengthy boring code
   //pseudocode instead:

   var words = extract_all_unique_words_from_the_post();
     //excluding punctuation
     //and other formatting details (markup).

   var tags = extract_tags_from_post();

   return were_the_tags_used_enough_in_post(words, tags);
}

Note: A more sophisticated algorithm here could help actually determine (and recommend) more appropriate tags based on word frequencies, but that’s beyond what I have time to implement for performance testing purposes. It would need to know to avoid common used words such as ‘the’, ‘code’, ‘error’, ‘problem’, ‘unsure’, etc (you get the point). It would then need to go further and know what words actually make sense to describe the technology (language/environment) the stack overflow question is about.

The Parallel operation is applied to a ‘Where’ filtering of data and this is where the timing and the reporting of the performance is based on. Making use of System.Diagnostics.StopWatch.

//running sequentially:
posts.Where(p => IsDescriptive(p));

// vs making use of parallel processing:
posts.AsParallel().Where(p => IsDescriptive(p));

On average this function making use of .AsParallel(), for varying records quantities from 100k to 300k would result in a 1.75 times speed up over the function operating sequentially on a single core. Which is what I was hoping to see.

All this was performed on a boot from VHD instance of Windows 7 (a great setup guide by Scott Hanselman here) with Visual Studio 2010 Beta 1 and SQL Server 2008, so I do understand there was some performance hit (both running as a VHD and having SQL on the same machine) but on average for effective PLINQ setup functions there was at least a 1.6 times factor speed up.

It’s that simple; it doesn’t do much yet, but there is potential for improved/more-interesting data analysis and performance measuring of it too. I will make time to clean up the demo application and post the solution files in a future post, so stay tuned for that. When I get a chance I’ll also try to investigate more of the data manipulations people are performing via data mining techniques and attempt to re-create them just for more performance tests. When I do I’ll be starting here.

That’s it, I’ll have a follow up post with some more details in particular the types of queries I had that did not benefit from PLINQ once I get a chance to determine how they were flawed or if they simply just run better on a single thread/core.

Due to social and sensor networks, it is estimated that data volume is doubling every 9 to 12 months. Analysis is required in realtime to derive knowledge from distributed databases. Awareness is improved by adding sources, e.g. the internet of things. The term data mining, reportedly coined by Robert Hecht-Nielsen a couple of decades ago, denotes automated fact-finding, knowledge discovery, rule inference and prediction activities. The field follows predecessors such as statistics, originally named for state demographics and economics, and machine learning. ACM dedicated a knowledge discovery and data mining group, KDD, in 1989.

In classic science, a hypothesis is often disproved by experiment, whereas in this case, tests yield a data-driven hypothesis. Patterns of interest are useful or novel, though most are not. More recently, the field picked up steam as analysis times for huge databases became excessive, disparate sources needed to be quickly connected and dimensionality, or number of attributes, expanded. These result in ways to assign meaning which leads to knowledge which is communicated as information assuming that errors are avoided or corrected. The result is better visualization and built-in database intelligence.

Government and security have been major proponents, e.g. for profiling. Other applications include biomed, insurance, physics, business intelligence, CRM, information retrieval, OLAP online analytical processing, text mining and analysis, finding experts, sports stats, and digital libraries. Besides software, tools include decision trees and neural networks. Models may be verified by splitting the data and verifying the equivalence of results on both parts.

Major tasks have been outlined as:

• classification, sequence detection, genetic algorithms, nearest neighbor, naive bayes classifier, logistic regression and discriminant analysis;
• affinity analysis, market basket, association analysis, rule learning, rough sets, and sequence detection;
• prediction, regression, and time series analysis forecasting;
• segmentation, cluster analysis, and kohonen networks;

A couple of the popular standards are CRISP-DM, cross industry standard process for data mining, and PMML, predictive model markup language.

There is plenty of software such as R, SAS SEMMA for sample explore modify model assess, SPSS, Netbase, Statistica, opensource Labkey, Rattle GNOME GUI, GNU octave, Weka-3, Apache Hadoop, Datalogic/R, Mozenda scraper. IBM, Oracle and Microsoft have offerings.

Other than usability, system integration and projections from prior knowledge, issues commonly revolve around privacy and performance. Congress has discussed consumer protections, though users are tracked from an increasing number of government social-network sites and cloud security standards are still in development. Data may be missing. Patterns may not be understandable. Noisy data can result in spurious patterns, though source correction is improving. Relationships between fields may be more complex than assumed.

Kdnuggets is a general resource site.

Also see bookmarks for media links.

JOLT é a sigla para Journal of Law & Technology da Universidade de Harvard. Isso é: coisa fina, vários PDFs, alguns deveras interessantes.

Nesse link tem um PDF sobre “DataMining & Antritust” e versa sobre o cruzamento de dados para criar uma pirâmide de preços. No abstract tem um exemplo com a Amazon:

In 2000, customers of Amazon.com discovered that the online retailer was varying the prices charged for DVDs depending on the identity of the purchaser.1 Although Amazon discontinued what it described as a “price test”2 after public outcry, Amazon’s brief foray into first-degree price discrimination stands as a noteworthy example of the possibilities for price discrimination using aggregated data. In its price test, Amazon sought to use information it already had about its customers to predict higher prices that the customers would still belikely to pay.

Die US-Softwarefirma EchoMetrix Inc hat sich auf Kinderschutz-Filtersoftware wie Sentry und FamilySafe spezialisiert. Damit können Eltern bestimmen, was ihre Kinder im Internet sehen können und was nicht. Im Juni stellte das Unternehmen einen weiteren Service vor:

Pulse. Hier können Unternehmen mit Produkten, die auf Kinder zielen, erfahren, was die Kinder so über diese Produkte denken. Weil EchoMetrix die Chats der Kinder belauscht, sobald ihre Filtersoftware auf deren Computern installiert ist, und deren Reaktionen zu den betreffenden Produkten bewertet.

EchoMetrix CEO Jeff Greene findet, das sei in Einklang mit US-amerikanischen Datenschutzgesetzen. Und ausserdem können sich besorgte Eltern nach der erfolgreichen Installation durch die Website des Anbieters klicken und dort in ihrem Account die “Weitergabe von Daten an vertrauenswürdige Geschäftspartner” ausklicken. Das steht aber nirgendwo, man muss es selber herausfinden.

Seht ihr? Sie machen Geschäfte mit unschuldigen Kindern. Und bespitzeln deren Unterhaltungen für lukratives Datamining. Was denkt ihr, was sollte man mit solchen Leuten machen?

( google news via boingboing)

Comment: Listen, you fake progressives and fake liberals. It wasn’t okay when Bush did it, SO WHY ARE YOU OKAY WITH OBAMA DOING IT?! You are not liberals. You are not progressives. You are neoliberals. You are the same as the fascist neocons who sucked Bush’s dick. So go ahead and suck Obama’s dick and choke on it.

——————————————————-

Obama White House Has Secret Plan To Harvest Personal Data From Social Networking Websites

Submitted by Ken Boehm on Mon, 08/31/2009 – 19:07
http://nlpc.org/stories/2009/08/31/obama-white-house-has-secret-plan-harvest-personal-data-social-networking-website

NLPC has uncovered a plan by the White House New Media operation to hire a technology vendor to conduct a massive, secret effort to harvest personal information on millions of Americans from social networking websites.

The information to be captured includes comments, tag lines, emails, audio, and video. The targeted sites include Facebook, Twitter, MySpace, YouTube, Flickr and others – any space where the White House “maintains a presence.”

In the course of investigating procurement by the White House New Media office, NLPC discovered a 51-page solicitation of bids that was filed on Friday, August 21, 2009. Filed as Solicitation # WHO-S-09-0003, it is posted at FedBizzOps.com. Click here to download a 51-page pdf of the solicitation.

While the solicitation specifies a 12-month contract, it allows for seven one-year extensions. It specifies no dollar cap. Other troubling issues include:

• extremely broad secrecy terms preventing the vendor from disclosing to the public or the media what information is being captured and archived (page 7, “Restriction Against Disclosure”)
• wholesale capturing of comments by non-White House staff on publicly accessible sites
• capturing of content of any type (text, graphics, audio, or video)
• capturing of comments by both Obama critics and supporters, with no restriction as to how the White House would use the information.

This is the third controversy involving the White House internet operations in less than a month. First, Obama’s New Media operation asked supporters to send information about critics of the White House health care effort to a White House email. This provoked a storm of criticism and the White House retreated. Then large number of people complained of getting email spam from the White House supporting the President’s health care position.  Again the White House was forced to back down.

Now the same people at the White House are at it again with an ambitious plan to harvest huge amounts of information from the web and specifically social networking sites.

Given the White House’s recent abuse of its New Media operations, this huge, new secretive program is yet another sign that this Administration is at best indifferent to privacy rights and at worst prepared to violate civil liberties for political purposes.

Perhaps anticipating negative reaction to the invasiveness of the plan, a justification is provided in a Q&A. section of the solicitation. Question #9 reads:

The Presidential Records Act does not require the storage or archiving of non-EOP content, as such is there a specific reason as to why the content provided on EOP related websites in the form of comments is included in these archiving procedures?

Answer: The PRA includes in its definition of presidential records content ―received by PRA components and personnel. Out of an abundance of caution, we are treating comments made by non-PRA personnel on sites on which a PRA component has a presence as presidential records, requiring them to be captured or sampled.

Of course, this interpretation of the Presidential Records Act is so expansive that virtually any communication mentioning the president or the Administration could become subject to collection and archiving under the Act. This is not out of an “abundance of caution,” but out of an over-abundance of power. President Obama should make sure that this plan goes no further.

—————————————————-

You can’t even get to NLPC’s website www.nlpc.org (National Legal Policy Center) to read their write up as I would assume it’s getting hammered.

You can only find it on various blogs at the moment. Below is a link to the contract description/PDF

http://nicedeb.wordpress.com/2009/09/01/report-obama-white-house-plans-to-harvest-personal-data-from-networking-sites/

This is the link to NLPC’s report/article but evidently their servers are down

http://nlpc.org/stories/2009/08/31/obama-white-house-has-secret-plan-harvest-personal-data-social-networking-website

contract description/PDF
https://www.fbo.gov/index?s=opportunity&mode=form&id=eec856940efb75b2b1c11e2b1d5660a4&tab=core&_cview=0&cck=1&au=&ck=

————————————————————

White House Still Trying To Get Information On Citizens
The National Legal and Policy Center (NLPC) has discovered a secret White House project to harvest personal date from social networking websites like facebook and twitter.

The White House office of New Media has sent out a request for proposals from technology vendors to develop and run the project. According to the proposal request, the information to be captured includes comments, tag lines, emails, audio, and video.

The targeted sites include Facebook, Twitter, MySpace, YouTube, Flickr and others — any space where the White House “maintains a presence.” The Proposal requests a bid covers allowing this project to last eight years.

In all fairness if you read the PDF of the solicitation, it speaks of the project as a way to to comply with the Presidential Records Act. But then there are the frightening parts especially for this administration which promises to be the most transparent in history. The disturbing parts of the proposal include:

• Extremely broad secrecy terms preventing the vendor from disclosing to the public or the media what information is being captured and archived (page 7, “Restriction Against Disclosure”)
• Wholesale capturing of comments by non-White House staff on publicly accessible sites
• Capturing of content of any type (text, graphics, audio, or video)
• Capturing of comment by both Obama critics and supporters, with no restriction as to how the White House would use the information.

http://www.rantburg.com/poparticle.php?D=2009-09-02&ID=278037&loc=interstitialskip

Введение

В этой статье я расскажу о состоянии на сегодняшнем российском рынке ноутбуков. Всю аналитику мы будем проводить с помощью кода на python. Думаю она будет полезна как тем, кто ищет ноутбук, так и тем, кто хочет потренироваться написанию на python.

Начнём

Для анализа нам необходим набор данных, к сожалению я не смог обнаружить веб-сервисы у российских он-лайн магазинов ноутбуков, поэтому мне пришлось скачать прайс-лист одного из них (я не стану называть его) и вытащить из него цены и основные параметры (по-моему мнению таковыми являются: частота процессора, диагональ монитора, объем оперативной памяти, размер жесткого диска и объем памяти на видео-карточке). Далее я провёл некоторый анализ по следующим вопросам:

1. Средняя стоимость ноутбука
2. Усредненные параметры железа на ноутбуках
3. Самая дорогая/дешевая конфигурация ноутбука
4. Какой из параметров конфигурации больше всего влияет на его цену
5. Прогнозирование цены указанной конфигурации
6. График распределения конфигураций и цен

Lets code

Прайс-лист, который мне удалось заполучить я сохранил в формате CVS, для работы с ним необходимо подключить модуль cvs:

Так же подключим модуль для работы со случайными числами и регулярными выражениями, которые в последствии нам понадобятся.

Далее создадим метод для чтения и получения ноутбуков:

здесь всё просто, мы читаем на файл с данными data.csv и фильтруем по результату функции create_notebook, т.к. не все позиции в прайсе являются ноутбуками, а вот кстати и она:

Как вы можете заметить, я решил не обращать внимания на вендора, модель и тип процессора (здесь конечно не всё так просто, но тем не менее), а и ещё – в данном методе присутствуют мои кастомные функции-помощники:

Хочу заметить, что писать для питона лучше всего в стиле наборов данных, а не ООП структур, в связи с тем, что язык больше располагает к такому стилю, однако для наведения некоторого порядка в нашей доменной области (ноутбуки), я ввёл класс, как вы могли заметить выше (notebook = Notebook())

Отлично, теперь у нас есть структура в памяти и она готова для анализа (2005 различных конфигураций и их стоимость), что же начнём:

Средняя стоимость ноутбука:

Исполняем код и видим, что 1K$, как стандарт для компьютера всё ещё в силе:

Усредненные параметры железа на ноутбуках

Та-да, и в наших руках усредненная конфигурация:

Самая дорогая/дешевая конфигурация ноутбука:

Функции идентичны, за исключением функций min/max

Какой из параметров конфигурации больше всего влияет на его цену

Очень интересно было бы узнать, за какой из параметров конфигурации мы платим больше всего денег. Прикинув, я предположил, что скорее всего это диагональ монитора и частота процессора, ну что же, думаю, что стоит проверить это.

Для начала наш набор параметров конфигурации стоит немного модифицировать. В связи с тем, что единицы измерения различных параметров различны в своём порядке, нам необходимо привести их к одному знаменателю, т.е. нормализовать их. Итак, приступим:

В данной функции я нахожу максимальные значения для каждого из параметров, после этого формирую результирующий список ноутбуков, в котором каждый из параметров представлен в виде коэффициента (его значение будет колебаться от 0 до 1), показывающего отношение его параметра к максимальному значению в наборе, к примеру память в 2048Mb даст конфигурации коэффициент в ram = 0.5 (2048/4056).

Вклад каждого из параметров мы будем считать в рублях, для наглядности, хранить эти веса мы будет в наборе:

Я предлагаю исчислять эти коэффициенты для каждой конфигурации, а после этого определить среднюю величину всех коэффициентов, что даст нам усредненные данные о весе каждого из элементов конфигурации.

Как же мы будет устанавливать коэффициенты для каждого элемента конфигурации? Мой способ заключается в следующем:

• нам необходимо в случайном порядке наращивать, либо уменьшать значение одного из коэффициентов
• после чего анализировать, приблизились ли мы к цене за конфигурацию, при умножении вектора параметров на вектор коэффициентов (напомню, что в нашем случае это рубли)
• если приближение состоялось, ты мы повторяем данное действие, если же нет, то отменяем его
• повторять данный порядок до той степени, пока не приблизимся к нашей цене с установленной нами точностью

Вот реализация данного алгоритма:

inc – переменная отвечающая за цвеличение/уменьшение коэффициента, способ её вычисления объесняется тем, что данное значение должно быть тем больше, чем больше разница в ошибке, для быстрого и более точного приближения к желаемому результату.

Умножение векторов для получения цены выглядит следующим образом:

Пришла пора выполнить анализ:

Данный набор мы получили, благодаря усреднению коэффициентов, полученных для каждой из конфигураций:

Итак, у нас получился желаемый набор, что же мы можем сказать из этих цифр, а можем мы составить рейтинг параметров:

1. Диагональ монитора
2. Объем жесткого диска
3. Частота процессора
4. Объем видео-карточки
5. Объем оперативной памяти

Хотелось бы отметить, что это далеко не идеальный вариант, и у вас могут получится иные результаты, однако, моё предположение, о том, что частота процессора и диагональ дисплея наиболее важные параметры в конфигурации, частично подтвердились.

Прогнозирование цены указанной конфигурации

Классно бы было, имея такой богатый набор данных, уметь прогнозировать цену на заданную конфигурацию. Этим мы и займемся.

Для начала преобразуем нашу коллекцию ноутбуков в список:

Далее нам понадобиться функция, способная определить расстояние между двумя векторами, хорошим вариантом я вижу функцию эвклидова расстояния:

Корень из суммы квадратов разностей довольно таки наглядно и эффективно показывает нам насколько один вектор различен от другого. Чем же полезна для нас данная функция? Всё просто, когда мы получим вектор, с интересующими нас параметрами, мы пробежимся по всей коллекции нашего набора и найдём ближайшего соседа, а его стоимость мы уже знаем, отлично! Вот как мы это сделаем:

Далее, можно немного усложнить задачу, а так же точность предоставляемых данных. Для этого мы введем функцию, использующую классификацию методом k взвешенных ближайших соседей:

взвешенных ближайших соседей – это метрический алгоритм классификации, основанный на оценивании сходства объектов. Классифицируемый объект относится к тому классу, которому принадлежат ближайшие к нему объекты обучающей выборки.

Ну и взять среднее значение среди некоторого количества ближайших соседей, что сведет на нет влияние цен вендора, либо специфичности конфигурации:

*последние 3 алгоритма взяты из книги Сегерана Тоби “Программируем коллективный разум”

И что же мы получаем:

Цены рыночные и этого вполне достаточно, хотя мы абсолютно не учитываем в этой реализации, к примеру частоту процессора и диагональ монитора (для этого нам необходимо добавить в функцию сравнения векторов их веса , которые мы вычисляли в предыдущем пункте)

График распределения конфигураций и цен

Хочется объять картину распределения целиком, т.е. нарисовать распределение конфигураций и цен на рынке. Ок, сделаем это.

Для начала надо поставить библиотеку matplotlib. Далее подключить её к нашему проекту:

Так же нам понадобится создать два набора данных, для оси абсцисс и ординат:

И функцию, в которой мы построим график распределения:

И что же мы получаем:

В завершение

Итак, у нас получилось провести небольшой анализ российского рынка ноутбуков, а так же немного проиграться с python.

Исходный код проекта доступен по адресу:

http://code.google.com/p/runm/source/checkout

This is a quick video outlining some ways in which the new Data for Research dataset service from JSTOR could prove beneficial to researchers.

I double dog dare you to search for some terms looking for their academic currency. When did pan-Africanism hit the scene? Sushi enter the academic vernacular? Want to know all the key terms from any of the 50 disciplines in the corpus? It is interesting stuff, especially considering the size and span of the database (14 billion words, 350 years, 5 million+ articles).

What’s Wrong With Fusion Centers – Executive Summary
Info Sheet
American Civil Liberties Union
December 5, 2007 [Or May 12th - it's unclear.]
Jay Stanley and Barry Steinhardt

“These new fusion centers, over 40 of which have been established around the country, raise very serious privacy issues at a time when new technology, government powers and zeal in the “war on terrorism” are combining to threaten Americans’ privacy at an unprecedented level.”

Centers Tap Into Personal Databases
Article
Washington Post
April 2, Wednesday 2008
Robert O’Harrow Jr.

“Intelligence centers run by states across the country have access to personal information about millions of Americans, including unlisted cellphone numbers, insurance claims, driver’s license photographs and credit reports, according to a document obtained by The Washington Post.”

So let me start with the following caveat:

I work for the Stor of J. There, I said it and I am proud of it.

That being said, this Data for Research service mentioned below is pretty amazing, especially if you love

• academic literature
• nerdy librarian things
• data/statistics/factoids to impress people at parties
• key terms/word counts/academic jargon/currency

JSTOR is offering a beta service called “Data for Research”. The original intention of the Data for Research tool was to make it easier to fulfill requests for data sets and support data mining needs. However, the DfR beta also makes it possible to search and browse across all JSTOR collections, using a type of faceted search interface. The journal content on this beta site is updated 1-2 weeks after each content release on the main site.

With DfR, researchers can

• conduct full-text and fielded searching of the entire JSTOR archive using a powerful faceted search interface. Using this interface one can quickly and easily define content of interest through an iterative process of searching and results filtering.
• view document-level data including word frequencies, citations, and key terms.
• request and download datasets associated with the content selected.

From DfR, you can also request and download datasets associated with selected content or automate this process with our API. Curious to know when academic vocabulary fell in and out of favor in academic circles? DfR lets you track that information from the over 14 billions words, 4.8 million+ articles and 350 years worth of academic research found in JSTOR.

Personally, I love the fact that the term perestroika peaked in academic literature, perhaps not surprisingly, in the early 1990’s, while tuberculosis seemed to gain some use as an academic term at the turn of the last century. Librarians should take note as well as DfR will basically automatically pull the key terms for each discipline over the entire corpus. If you are struggling with synonymous search terms for an intricate advanced search statement, this is the place to go. Click on any of the 50 disciplines and see all the key terms associated with it.

A special feedback form has been established for this project (http://dfr.jstor.org/requests/contact/) and is linked to all of the pages of the DfR site. If you can think of any ways you want to mine the JSTOR data not supported in this beta, let JSTOR know and they will try to incorporate it into the next instance of DfR.

Beeindruckend. Matthew Hurst hat in seinem Blog einige Blogosphären Visualisierungen veröffentlicht. Mein persönlicher Favorit ist die folgende:

via http://datamining.typepad.com/

Dazu schreibt Matthew:

The dark edges show the reciprocal links (where A has cited B and B has cited A), the lighter edges indicate a-reciprocal links. The larger, denser area of the graph is that part of the blogosphere generally characterised by socio-political discussion (the periphery contains some topical groupings). Above and to the left is that area of the blogosphere concerned with technical discussion and gadgetry.

Mehr Visualisierungen gibt es in seinem Artikel Science July 24th 2009.

-r-

Clinical trials means a pharmaceutical company tests their new product on ‘human guinea pigs’, in a more or less closed  and controlled environment.

EHRs would allow much more data to be accumulated in a more real-world setting: thousands or tens of thousands of patients of various ages, races, genders etc using a medication in the real-world.

Any companies out there doing data-mining of EHRs or EMRs? At the HIE level perhaps?

Here is a story from the Department of Health and Human Services proposing to use EHRs (scrubbed of private patient data) to improve the health of minorities.