Data Analysis with Open Source Tools: A Hands-On Guide for Programmers and Data Scientists

I'm a Python software developer with an interest in applied statistics. This is an excellent book on data analysis, but for review purposes, it's worth initially pointing out what this book is not.It is not a comprehensive survey of open source tools that are available, and it does not contain many examples of working code to implement the techniques he talks about, though there are some. For this reason, I'd strike the "with Open Source Tools" from the title in evaluating whether you want to purchase the book.The author greatly favors mathematical notation over code examples in describing the data analysis techniques he presents. While this is not a bad thing per se, you'll have to struggle to comprehend the content if you're a programmer without an academic familiarity with math, or if you've been away from mathematics for a long time.As other reviewers have pointed out, the organization of the content is somewhat disjointed. Going from chapter to chapter, there is little in the way of causality, and the early chapters are pretty math-heavy. The reader is advised to consult appendices at the back of the book to refresh themselves on the basics, if required.Wait! I didn't say you shouldn't buy it.Despite a few shortcomings, this book does offer a good introduction and overview of several basic techniques. It's an excellent survey of the current data analysis landscape for anyone who's not familiar with it. If a topic seems irrelevant to you, it's pretty easy to skip that chapter and move forward.On top of that, the author's writing style and ways of explaining relatively esoteric concepts is generally very good. As with many good books, you get the sense the author is a co-worker, trying to explain something to you in terms you can understand. It's very example-based, even if those examples don't always involve code.All in all, to get the most out of this book, the best approach is careful and methodical study. The author covers many topics quickly, and not any one in depth, so if one chapter interests you, I'd plan on consulting other resources on particular topics. Luckily, the author does offer several "Further Reading" recommendations for each topic.Most books containing information on these techniques are far harder to read, and they generally cost at least twice as much. Highly recommended. Thanks for this one, Philipp.

I bought this book hoping for a reference on open source tools. But the open source tools are a minor aspect of this book. The core is about data analysis--and it is fantastic. I should have known this from the title I suppose: the "data analysis" is in big font with a colorful background, and "with open source tools" is in small font--and it is literally about the same ratio with the book. Each chapter has small section that works one example with an open source tool. And there is a chapter at the end about the array of open source tools available.But the data analysis aspects of the book--most outstanding. I have a master's in computer science, and do data and analytics for a living, so I have many books on the topic. Some books with more of a theoretical and rigorous foundation, some with more of a hands-on slant. I was expecting this book to be the latter, but it is quite the former.Yet it is still very practical. It is not a "theory" work as such, just a rigorous book useful in practice (there is a big difference!). Throughout the book the author points out the value of solving the problem at hand, rather than being excessively precise--which is the bigger risk in this domain. Examples would be: using visuals to get a feel for data but not trying to use visuals to give precise answers (which they fundementally cannot), and using techniques that get "close enough" such as perturbation.And it is extremely well written. The writing is in reasonably simple English, relative to the topic, yet not insulting or goofy the way the "Dummies" series can be for example. It is easy to read yet content rich--a fantastic combination.

I've read some of the other reviews, and I do agree with most of the criticisms. There are quite a few errors in formulas and in the text, and it would've been really nice if the source codes and data files were given in a CD or were available on a website.That being said, the book addresses a lot of different topics - ranging from the introductory, freshman-level statistics to more advanced data mining and machine learning techniques, and passing through notions of design. It doesn't go in depth into each of them, but offers a fairly good overview, and references in case you're interested. Furthermore, the author gives some useful hints on how to do outside-the-box thinking and how to apply these techniques into business.Being a physics grad student, I've found many of the topics pretty much basic, but even so, I've learned a lot. Overall, a great introduction; I really hope the flaws are corrected on a future 2nd edition.

Data Analysis with Open Source Tools does a great job covering a lot of topics in way that balances theoretical explanations and practical demonstration. In keeping true to its title, a wealth of tools (and data sources) are identified and explored.Because the book offers a balance between explanation and demonstration it can be read in two different ways. First, you can read the chapters without getting involved with the code to get a better understanding of the whys and hows of the different analysis techniques. On the other hand, if you are more of a brass tacks person, you can focus on the code, run the examples, and just skim the explanations.For those that are exploring the world of data analysis, this book is a great compliment to Segaran's Programming Collective Intelligence: Building Smart Web 2.0 Applications and Russell's Mining the Social Web: Analyzing Data from Facebook, Twitter, LinkedIn, and Other Social Media Sites. Where the books overlap the explanations and examples differ which helps enormously when trying to master the concepts and techniques. However, each book contains topics not in the others. Collectively they offer a rather powerful set of tools.Having read the other books prior to this one, I really appreciated the time spent on the mathematics behind each technique. The others get your hands dirty very quickly - and I appreciated that greatly when first exploring data mining - but I found myself wanting to have a deeper understanding which this book so nicely provides. As Janert mentions in the first chapter, the succinct notation of mathematics is much clearer than having to try to extract the essence of twenty lines of source code. Without a doubt, though, Data Analysis is dense which and that might turn a few people off.All said and done, I'm glad I took the time to read the book and will definitely keep it nearby.

I don't expect much when I brought the book. Part of it is that its title is a little bit vague. But it turned out that I learned quite a lot from it. Many books contain materials that one can easily find on the internet free. This book is different. It is not a reference book nor a textbook that goes through some subject systematically. It contains topics that are quite diverse and incoherent and is from the author's personal experience. However, it does not decrease its value. Instead, it adds value and makes it unqiue. It is not dry and do not filled with so-called "humor" or useless generic statements about the field. It is not hard to read. In fact it feels more that a peronsal journal in data analysis that the author walk with you and share his experience with you. When you finish the journey with him, you pick up a bunch of useful ideas and techniques in data analysis that you can use in your own work. Highly recommended!!!

While I'm not an expert in all the areas covered in this book, I am in a few. In those areas, this book is really wrong -- actually doing damage wrong.For instance, when talking about regressions, the author claims that:1) "Regression only makes sense when you want to use it as a prediction." This a very wrong. Any decent Econometrics book well be almost entirely about counter examples.2) "Linear regression is appropriate only if the data can be described as a straight line." The "linear" in linear regression doesn't mean that at all. It just means that the form of the function must be linear in the coefficients to be calculated. In particular, x = a + b*x + c*x^2 will fit a parabola to the data.3) "Historically, one of the attractions of linear regression has been that it is easy to calculate." It's easy to calculate for a single independent variable but multivariate regression are devilishly difficult to calculate because of numerical issues.The is where I really have problems because multiple regression models are one of the most useful techniques available for understanding the effect of different factors and the author just dismisses them out of hand.There are other problems:When talking about the CDF, he defines it as the integral of the histogram. The histogram is not the probability density function. The PDF is defined to integrate to 1 where the histogram integrates to something else.The formula for standard deviation is wrong, the formula for exponential moving average is wrong (a typesetting problem).So, my problem is that I find a lot of problems with the portions of the book I know. Can I trust the remainder of the book or should I be wary? In this case, I'm wary.

I love this book on data analysis, but I do understand not everybody likes this style.From a theoretical physics background, I appreciate the book and the author a lot. The writer put a lot of effort in explaining the background on each topic from the perspective of someone who knows a bit about the topic but not in depth. People who are currently data scientists are from different technical background, and the text is a good introduction into the topics. Technical details are not overwhelming, which is good for people who can pick up the technicalities on their own through other books and the web.If one is looking for the open source tools implementation, he is certainly disappointed. (The title of the book is unfortunately misleading.) If one is looking for technical details, this is not a good option for them. However, to gain the insights and the big picture, this is the best book.The following chapters are well written:- Chapter 2 (A Single Variable: Shape and Distribution): This brings people into the style of the book, some basics to data analysis and wrangling, and an introduction to NumPy.- Chapter 8 (Models from Scaling Arguments): Mathematical modeling to data, something a lot of theorists doing!- Chapter 9 (Arguments from Probability Models).- Chapter 13 (Finding Clusters): Introduction to various clustering (unsupervised learning) techniques.- Chapter 18 (Predictive Analytics): Something hot recently. This serves a good piece of introduction to the big picture because a lot of other books are overwhelming with the technical details that we often get lost when working with these tools.

Book Review--Data analysis with open source tools (Oil IT Journal ([...]), December 2010)A new book from O'Reilly Media provides insight into statistics--targeting the `risk area' of using statistical concepts with `a limited understanding of what they really mean.'Philipp Janert's new book `Data analysis with open source tools,' (DA) described as a `hands-on guide for programmers and data scientists' is more than that. Janert sets-out to address two common data analysis `risk areas.' One is the use of statistical concepts `with a limited understanding of what they really mean.' The other is the deployment of `complicated, expensive black box solutions' used in the place of a `simple, transparent approach.'One scene setting anecdote involved a client whose IT department recommended a cluster-based neural net to analyze product defect data. The solution Janert found was rather more economical--a one line calculation--not even a code! Janert offers some useful advice on educating the customer--observing that `few clients are in a good position to ask meaningful questions.' This inevitably means that the statistician needs to be cognizant of the client's business and terminology.This book offers a deep approach to real world tasks. There is much more text than code. Janert's contention that `statistics is usually equated with a college class that made no sense at all' will chime with many. His promise to `explain what statistics really is' should excite. He also provides insight into computational issues. For instance the value of the sine and cosine function for large values of x eventually degenerates to a random number as the limit of a float's resolution nears.While the thrust of DA is solving business (rather than scientific) problems, Janert is a polymath who is interested in his subject. This is conveyed particularly in the chapter on classical statistics which includes exposés on significance, design of experiment and a fascinating section on the Bayesian and frequentist approaches that are pretty a propos to the seismic imaging folks as we learned last month during the SEG's Albert Tarantola memorial.Also of interest to the oil and gas community is the section on financial analysis with clear explanations of net present value, risk analysis and opportunity costs.Janert's tools of choice are Python (with NumPy and SciPy) and Unix. On which topic, Janert curiously saves some unequivocal advice for page 494, `Work on Unix--I mean it. Unix was developed for precisely the kind of ad-hoc programming [...] that encourages you to devise solutions.'It is hard to fault this book except perhaps that at nearly 500 pages, it is too short! The couple of pages on Map/Reduce, for instance, fall way short of Janert's pedagogical aims. But quibbles apart, DA gets a double thumbs up from this reviewer.

The book provides very good math and stats foundation without diving into the code. I liked the methodology behind the way the book is structured.

I have to agree with a lot of the US reviews. I am missing a focus in the book.The author wants to make a point how important it is to understand the math behind real world problems, but I was disappointed by his attempts to convey mathematical principles. Formulas may work for some people, to me the book failed to point out why they are necessary - or how i can add value with them in the analyses i do. In this regards, the author overpays his dues to his academic background. I can see how the author studied physics and addresses people with like-wise framed minds. But for these people, the book will be too trivial. The major disappointment for me was that the book failed to live up to its expectations regarding the subtitle "with open Source tools". I would have expected a range of cool tools to work with, instead it's GNU and R, and there is not a single end-to-end case of getting the data, figuring out the issue and then presenting it in a graph. Sometimes, the style is too conversational, sometimes it is too strict and abstract. There are few moments when the two extremes touch. Other parts of the book - were the author shares his academic insights - felt awkward. The statement "You will never understand what mathematics is if you see it only as something you use to obtan certain results" will definitely find its way in my "Dictionary of Received Ideas".Still after all this negative criticism, I am giving it an average 4 stars. Why? There were some conversational parts that are helpful. This happens especially when the author highlights pitfalls and real-world application on distribution laws and showing/interpreting graphical analysis (although he doesn't point out how it's done). I can put these ideas to use, and they are valuable, because they show the true expertise of the author and can serve as a guideline for people learning to get familiar with advanced statistical analysis. And I want to give credit to the broad scope of the book. I prefer this to textbooks that focus on one aspect only. Although the book is often too abstract, I appreciate the approach to cover many topics in 10-20 page essays.

Every person involved in any computational scienceshould have read this book and always keep itat arm's reach.

This is the book you want, if you try to get quickly into scientific programming and visualization with Python and R! I strongly reccommend this book!

Un libro da avere per chi (come me ) si occupa di data analysis con strumenti open source ( spesso sviluppati per l'occasione).Un testo pregno di teoria ma con un occhio sempre orientato all'applicazione concreta della stessa.Davvero ben scritto,ben organizzato e spesso utile!Consigliatissimo!