People said Python is slow, how slow it can be

Whenever there is a programming speed competition, Python usually goes to the bottom. Some said that is because Python is an interpretation language. All interpretation language is slow. But we know that Java is also a kind of language, its bytecode is interpreted by JVM. As showing in this benchmark, Java is much faster than Python.

Here is a sample that can demo Python’s slowness. Use the traditional for-loop to produce reciprocal numbers:

import numpy as np
np.random.seed(0)
values = np.random.randint(1, 100, size=1000000)
def get_reciprocal(values):
    output = np.empty(len(values))
    for i in range(len(values)):
        output[i] = 1.0/values[i]
%timeit get_reciprocal(values)

The result:

3.37…

Azure Machine Learning(AML) provides a powerful and convenient way to build, deploy and run your ML applications. AML allows you to keep your established code build upon Scikit-Learn, PyTorch, or any other ML framework. This is nice.

What is more, if you test and run your experiments in your local machine, the cost is also pretty low. Based on my test, without firing up the cloud compute instance, you can use AML as low as $0.25/per day.

But, it seems not easy to set up the environment correctly. After many hours of trial and time wasted, I decided to document…

Plan work for next week, only

Delaying and deferring planned work to next release, sounds familiar?

“As Wes had predicted, more and more work is being deferred to the next release, including almost all of the testing.” — Chapter 6

Some years ago, when I was first time promoted as a dev leader and asked to start a project from scratch. My then manager asked me in the planning meeting: “Andrew, could you estimate the efforts and cost for list features?”

I was excited and confidently scan through the list. Without thoughtful contemplating, replied: “Feature A is relatively easy, my team should be able to work…

Are you fear of missing out on new and cool technologies? fear of checking out the wrong skill tree and miss the whole new industry and career?.

So, you tried to get familiar with all new fancy technologies. when Web 2.0 is hot, you learned Ajax; When Big Data is shooting on your ears, you start learning Hadoop, ok, Hadoop is slow, so you learned Spark; When experts told you Cloud ere is coming, you ditched everything and think that Docker will be the future, wait, Kubernetes is the real future…

In school, you learned C and C++, in a…

The feature selection problem

Machine Learning models are amazing when trained with an appropriate set of training data. ML models described in the textbook and using datasets from Scikit-learn, sample datasets are already carefully curated and effective for targeting ML models.

The problem is, in real life, training data could be a mass and not fit for any models at all. For example, you want to train a model to predict the next month's revenue based on current 100+ customer properties, like age, location, country, active visits #, last time visits time… on and on.

Some of those properties may useful, some could be…

After one crazy week of working on a Databricks project, I made a lot of mistakes and hence learned a lot. Here are some tips to share on how to make those mistakes I made.

Use Concatenated Spark SQL string in functions

With PySpark, we can either query a Spark Dataframe with Spark SQL or DataFrame DSL(domain-specific language).

The Spark SQL way:

# create a view from spark dataframe 
sdf.gud.createOrReplaceTempView('sdf_view')
# define you sql query as a string
sql_string = "select * from sdf_view"
# execute the spark SQL
result_df = spark.sql(sqlQuery = sql_string)

With Dataframe DSL, you can query the data without creating any views, almost…

Compare with machine learning models like Neural Network, I thought Decision Tree Classifier should be the most simple one. But I was wrong, this model is a bit complex than I thought. And the model also lands the foundation for other advanced models like LightBGM and Random Forest Decision Tree. So, I spent some time learning it and try to figure out how Decision Tree Classifier works.

How decision tree works

The model works very much like how a human mind classifying objects in the real world.

In the previous short article Track Dogecoin Real-Time Price with Python I leveraged Python’s requests and BeautifulSoup package to scrape the web HTML to grab real-time Dogecoin (or any other cryptos trade in Robinhood.com).

My holding number of Dogecoin is like the coin itself, is a joke. The main purpose is not for trading but to get hand dirty and see how I can use Python to scrape the web with minimum lines of code, and it looks working pretty well.

The next question follows: How can I get the historical price information in daily or even hourly granularity for…