- Introduction to Pandas in Python<\/a>
- Versatile Data Types<\/a><\/li>
- Data Cleaning and Transformation<\/a><\/li>
- Compatibility with Libraries<\/a><\/li><\/ul><\/li>
- Top Pandas Data Exploration Functions<\/a><\/li><\/ul><\/div>\n\n\n\n
Introduction to Pandas in Python<\/h2>\n\n\n\n
Pandas in Python serves as a bridge between the raw data you have and the insights you seek. Think of it as a data powerhouse that enables you to clean, transform, and explore your datasets with unmatched ease. There are two fundamental data structures at the core of Pandas library: Series<\/strong> and DataFrame<\/strong>. A Series is like a column in an Excel spreadsheet, it holds data of a single type, such as numbers or strings. DataFrame, on the other hand, is just like the entire spreadsheet, a two-dimensional table containing multiple columns and rows. Here are some benefits Pandas offers:<\/p>\n\n\n\n
Versatile Data Types<\/strong><\/h3>\n\n\n\n
DataFrame can combine a wide range of data types within a single structure, including integers, floats, strings, and even more complex data structures like lists and dictionaries.<\/p>\n\n\n\n
Data Cleaning and Transformation<\/strong><\/h3>\n\n\n\n
By converting the dataset into Pandas’ DataFrame format, we could utilize functions provided by Pandas to perform data manipulation and apply mathematical operations. <\/p>\n\n\n\n
Compatibility with Libraries<\/h3>\n\n\n\n
Pandas is compatible with a wide range of Python libraries and tools, making it an integral part of end-to-end data analysis workflows. For example, it seamlessly integrates with popular visualization libraries like Matplotlib and Seaborn. <\/p>\n\n\n\n
Community and Documentation<\/p>\n\n\n\n
Most importantly, it has a huge community, which means there is a wealth of resources, tutorials, and documentation available for learning and troubleshooting. <\/p>\n\n\n\n
Top Pandas Data Exploration Functions<\/h2>\n\n\n\n
In this section, I will show some useful Pandas functions on an online store’s sales dataset. <\/p>\n\n\n\n
ID<\/th> Product<\/th> Category<\/th> Price<\/th> Quantity<\/th><\/tr><\/thead> 1<\/td> Laptop<\/td> Electronics<\/td> 800<\/td> 10<\/td><\/tr> 2<\/td> Tablet<\/td> Electronics<\/td> 300<\/td> 15<\/td><\/tr> 3<\/td> Shirt<\/td> Clothing<\/td> 25<\/td> 50<\/td><\/tr> 4<\/td> Jeans<\/td> Clothing<\/td> 40<\/td> 30<\/td><\/tr> 5<\/td> Book<\/td> Books<\/td> 15<\/td> 100<\/td><\/tr> 6<\/td> Headphones<\/td> Electronics<\/td> 50<\/td> 25<\/td><\/tr> 7<\/td> Dress<\/td> Clothing<\/td> 35<\/td> 40<\/td><\/tr> 8<\/td> Phone<\/td> Electronics<\/td> 600<\/td> 8<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n # Python code to generate the online store sales \nimport pandas as pd\n\ndata = {\n 'ID': [1, 2, 3, 4, 5, 6, 7, 8],\n 'Product': ['Laptop', 'Tablet', 'Shirt', 'Jeans', 'Book', 'Headphones', 'Dress', 'Phone'],\n 'Category': ['Electronics', 'Electronics', 'Clothing', 'Clothing', 'Books', 'Electronics', 'Clothing', 'Electronics'],\n 'Price': [800, 300, 25, 40, 15, 50, 35, 600],\n 'Quantity': [10, 15, 50, 30, 100, 25, 40, 8]\n}\n\ndf = pd.DataFrame(data)<\/code><\/pre>\n\n\n\n
\n\n\n\n1.
head()<\/code> andtail()<\/code><\/strong><\/p>\n\n\n\n- \n
head()<\/code>: Returns the first few rows of a DataFrame. By default, it shows the first five rows, but you can specify the number of rows you want to display.<\/li>\n\n\n\ntail()<\/code>: Returns the last few rows of a DataFrame. Likehead()<\/code>, it shows the last five rows by default.<\/li>\n<\/ul>\n\n\n\n# Display the first few rows of the DataFrame\nprint(\"First 3 rows:\")\nprint(df.head(3))<\/code><\/pre>\n\n\n\nID<\/th> Product<\/th> Category<\/th> Price<\/th> Quantity<\/th><\/tr><\/thead> 1<\/td> Laptop<\/td> Electronics<\/td> 800<\/td> 10<\/td><\/tr> 2<\/td> Tablet<\/td> Electronics<\/td> 300<\/td> 15<\/td><\/tr> 3<\/td> Shirt<\/td> Clothing<\/td> 25<\/td> 50<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n # Display the last few rows of the DataFrame\nprint(\"Last 3 rows:\")\nprint(df.tail(3))<\/code><\/pre>\n\n\n\nID<\/th> Product<\/th> Category<\/th> Price<\/th> Quantity<\/th><\/tr><\/thead> 6<\/td> Headphones<\/td> Electronics<\/td> 50<\/td> 25<\/td><\/tr> 7<\/td> Dress<\/td> Clothing<\/td> 35<\/td> 40<\/td><\/tr> 8<\/td> Phone<\/td> Electronics<\/td> 600<\/td> 8<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n2.
info()<\/code><\/strong><\/p>\n\n\n\ninfo()<\/code><\/strong> provides a concise summary of a DataFrame, including data types, non-null counts, and memory usage. This is useful to quickly understand the structure of your dataset.<\/p>\n\n\n\n# Get a summary of the DataFrame\nprint(\"DataFrame summary:\")\nprint(df.info())<\/code><\/pre>\n\n\n\n#<\/th> Columns<\/th> Non-Null Count<\/th> Dtype<\/th><\/tr><\/thead> 0<\/td> ID<\/td> 8 non-null<\/td> int64<\/td><\/tr> 1<\/td> Product<\/td> 8 non-null<\/td> object<\/td><\/tr> 2<\/td> Category<\/td> 8 non-null<\/td> object<\/td><\/tr> 3<\/td> Price<\/td> 8 non-null<\/td> int64<\/td><\/tr> 4<\/td> Quantity<\/td> 8 non-null<\/td> int64<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n3.
describe()<\/code><\/strong><\/p>\n\n\n\ndescribe()<\/code><\/strong> generates descriptive statistics for numerical columns in a DataFrame. It provides information like mean, standard deviation, minimum, maximum, and quartile values.<\/p>\n\n\n\n# Generate descriptive statistics for numerical columns\nprint(\"Descriptive statistics:\")\nprint(df.describe())<\/code><\/pre>\n\n\n\n<\/th> ID<\/th> Price<\/th> Quantity<\/th><\/tr><\/thead> count<\/td> 8.00000<\/td> 8.00000<\/td> 8.000000<\/td><\/tr> mean<\/td> 4.50000<\/td> 233.12500<\/td> 34.750000<\/td><\/tr> std<\/td> 2.44949<\/td> 307.38456<\/td> 30.127112<\/td><\/tr> min<\/td> 1.00000<\/td> 15.00000<\/td> 8.000000<\/td><\/tr> 25%<\/td> 2.75000<\/td> 32.50000<\/td> 13.750000<\/td><\/tr> 50%<\/td> 4.50000<\/td> 45.00000<\/td> 27.500000<\/td><\/tr> 75%<\/td> 6.25000<\/td> 375.00000<\/td> 42.500000<\/td><\/tr> max<\/td> 8.00000<\/td> 800.00000<\/td> 100.00000<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n Note: Non-numeric data cannot be shown in the table.<\/p>\n\n\n\n
\n\n\n\n4. value_counts()<\/strong><\/p>\n\n\n\n
value_counts()<\/code><\/strong> returns a Series with the count of unique values in a specified column. It’s particularly useful for understanding the distribution of categorical data.<\/p>\n\n\n\n# Count the occurrences of each category in the 'Category' column\nprint(\"Value counts for 'Category' column:\")\nprint(df['Category'].value_counts())<\/code><\/pre>\n\n\n\nCategory<\/th> Count<\/th><\/tr><\/thead> Electronics<\/td> 4<\/td><\/tr> Clothing<\/td> 3<\/td><\/tr> Books<\/td> 1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n5.
groupby()<\/code><\/p>\n\n\n\ngroupby()<\/code><\/strong> splits the data into groups based on a specified criterion, typically a column. This function is often followed by an aggregation operation, likesum()<\/code>,mean()<\/code>, orcount()<\/code>, to compute summary statistics within each group.<\/p>\n\n\n\n# Group the data by 'Category' and calculate the total quantity in each category\nprint(\"\\nGrouped by 'Category' with total quantity:\")\nprint(df.groupby('Category')['Quantity'].sum())<\/code><\/pre>\n\n\n\nCategory<\/th> Quantity<\/th><\/tr><\/thead> Books<\/td> 100<\/td><\/tr> Clothing<\/td> 120<\/td><\/tr> Electronics<\/td> 58<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n6.
pivot_table()<\/code><\/p>\n\n\n\npivot_table()<\/code><\/strong> creates a pivot table that summarizes data in a more structured format. You can define rows, columns, and values to create a multidimensional summary of the data.<\/p>\n\n\n\n# Create a pivot table to show average price for each category\npivot_table = df.pivot_table(index='Category', values='Price', aggfunc='mean')\nprint(\"Pivot table for average price:\")\nprint(pivot_table)<\/code><\/pre>\n\n\n\nCategory<\/th> Price<\/th><\/tr><\/thead> Books<\/td> 15.00000<\/td><\/tr> Clothing<\/td> 33.33333<\/td><\/tr> Electronics<\/td> 437.50000<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n7.
isnull()<\/code> <\/p>\n\n\n\nisnull()<\/code> <\/strong>returns a DataFrame of the same shape as the input, withTrue<\/code> for missing values (NaN) andFalse<\/code> otherwise.<\/p>\n\n\n\n# Check for missing values\nprint(\"\\nMissing values check:\")\nprint(df.isnull())<\/code><\/pre>\n\n\n\n<\/th> ID Product<\/th> Product <\/th> Category <\/th> Price<\/th> Quantity<\/th><\/tr><\/thead> 0<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 1<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 2<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 3<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 4<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 5<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 6<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr> 7<\/td> False<\/td> False<\/td> False<\/td> False<\/td> False<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n You can also combine
isnull()<\/code> withsum()<\/code> functions for a more comprehensive view.<\/p>\n\n\n\n# return the count of null value in every column \nprint(df.isna().sum())<\/code><\/pre>\n\n\n\nID<\/td> 0<\/td><\/tr> Product<\/td> 0<\/td><\/tr> Category<\/td> 0<\/td><\/tr> Price<\/td> 0<\/td><\/tr> Quantity<\/td> 0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n8.
fillna()<\/code><\/p>\n\n\n\nfillna()<\/code><\/strong> replaces missing values in a DataFrame with specified values or uses certain filling methods, such as mean, median, and mode (mostly for categorical features).<\/p>\n\n\n\n# Fill missing values in 'Price' with the mean price\ndf['Price'].fillna(df['Price'].mean(), inplace=True)\nprint(\"DataFrame after filling missing values:\")\nprint(df)<\/code><\/pre>\n\n\n\nSince the table doesn’t contain any missing value, it will return the exact same table.<\/p>\n\n\n\n
\n\n\n\n9.
plot()<\/code><\/p>\n\n\n\nplot()<\/code><\/strong> creates basic plots like line, bar, scatter, and more directly from a DataFrame. It simplifies the process of generating visualizations to aid in data exploration.<\/p>\n\n\n\nimport matplotlib.pyplot as plt\ndf.groupby('Category')['Quantity'].sum().plot(kind='bar')\nplt.title('Total Quantity by Category')\nplt.xlabel('Category')\nplt.ylabel('Total Quantity')\nplt.show()<\/code><\/pre>\n\n\n\n![\"Pandas](\"https:\/\/i0.wp.com\/datascihubs.com\/wp-content\/uploads\/2023\/08\/Pandas-Plot-min.png?resize=448%2C336&ssl=1\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Pandas in Python is a widely used library for data scientists. It emerges as a foundation library for data manipulation and analysis. If you are contemplating working with any type of data, Pandas is the go-to tool for data cleaning, transforming, and exploring complex datasets. In this article, we will see the concepts of data […]<\/p>\n","protected":false},"author":1,"featured_media":331,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[28,47],"class_list":["post-323","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-information","tag-data-analysis","tag-python"],"blocksy_meta":[],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/datascihubs.com\/wp-content\/uploads\/2023\/08\/Feature-Image-Python-min.png?fit=1280%2C720&ssl=1","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/posts\/323","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/comments?post=323"}],"version-history":[{"count":1,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/posts\/323\/revisions"}],"predecessor-version":[{"id":833,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/posts\/323\/revisions\/833"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/media\/331"}],"wp:attachment":[{"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/media?parent=323"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/categories?post=323"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascihubs.com\/index.php\/wp-json\/wp\/v2\/tags?post=323"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Data Cleaning and Transformation<\/a><\/li>
- Versatile Data Types<\/a><\/li>