Back ArrowExplore all kits

How to Fill In Missing Data Using Python Pandas

share link

by vsasikalabe dot icon Updated: Feb 6, 2023

technology logo
technology logo

Solution Kit Solution Kit  

Missing Data can occur when no information is provided for particular items. generally missing data is a common problem in real-life scenarios. Missing Data can also be used as NA(Not Available) values in pandas. In the data frame, some datasets simply arrive with missing data, either because it exists or was not collected.


Data is represented by two values:

  • None is a no value (nothing) that is often used for missing data in Python code.
  • NaN (Not a Number) is a special floating-point undefined or unrepresentable value.

In Pandas, None and NaN are used for indicating missing or null values. There are more useful functions for representing null values in Pandas DataFrame :

  • isnull()-The isnull() method returns a Boolean value True for NULL values, and otherwise False in the data frame.
  • not null()-not null is one of the functions of the pandas library in Python that detects if values are not missing.
  • drop na()-The drop na() method deletes the rows that contain NULL values.
  • fill na()-The NULL values are replaced by some specified value.
  • replace()-The replace() method replaces a specified value with another specified value.
  • interpolate()-It is used to estimate unknown data values between two known data values.


Here is an example of how to fill in missing data using python pandas:

Preview of the output that you will get on running this code from your IDE.

Code

In this solution we used pandas and numpy library of python.

filling out missing values in pandas dataframe

PythondotLines of Code : 40dotLicense : Strong Copyleft (CC BY-SA 4.0)

Dependent Libraries : 
df1 = pd.DataFrame({
        'FirstName':list('abcdef'),
        'LastName':list('aaabbb'),
         'PhNo':[7,np.nan,9,4,np.nan,np.nan],
         'uniqueid':[5,3,6,9,2,4],

})

print (df1)
  FirstName LastName  PhNo  uniqueid
0         a        a   7.0         5
1         b        a   NaN         3
2         c        a   9.0         6
3         d        b   4.0         9
4         e        b   NaN         2
5         f        b   NaN         4

df2 = pd.DataFrame({
         'PhNo':[10,90,30,20],
         'uniqueid':[3,6,9,4],

})
print (df2)
   PhNo  uniqueid
0    10         3
1    90         6
2    30         9
3    20         4

s = df2.set_index('uniqueid')['PhNo']
df1['PhNo'] = df1['PhNo'].fillna(df1['uniqueid'].map(s))
print (df1)
  FirstName LastName  PhNo  uniqueid
0         a        a   7.0         5
1         b        a  10.0         3
2         c        a   9.0         6
3         d        b   4.0         9
4         e        b   NaN         2
5         f        b  20.0         4

Instructions

Follow the steps carefully to get the output easily.

  1. Download and Install the PyCharm Community Edition on your desktop.
  2. Install pandas on your IDE from python interpreter in setting options.
  3. Create new python file on your IDE.
  4. Copy the snippet using the 'copy' button and paste (line no 1 to 9,line no 18 to 23 and line no 29 to 31) it in your python file.
  5. import the pandas and numpy library.
  6. Run the current file to generate the output.


I hope you found this useful. I have added the link to dependent library, version information in the following sections.


I found this code snippet by searching for ' Filling out missing values in pandas dataframe' in kandi. You can try any such use case!

Environment Tested

I tested this solution in the following versions. Be mindful of changes when working with other versions.

  1. PyCharm Community Edition 2022.3.1
  2. The solution is created in Python 3.11.1 Version
  3. pandas 1.5.2 Version
  4. numpy 1.24.1 version


Using this solution, we can filling out missing values in pandas dataframe.This process also facilities an easy to use, hassle free method to create a hands-on working version of code in python which would help us to filling out missing values in pandas dataframe.

Dependent Libraries

pandasby pandas-dev

Python doticonstar image 38689 doticonVersion:v2.0.2doticon
License: Permissive (BSD-3-Clause)

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

Support
    Quality
      Security
        License
          Reuse

            pandasby pandas-dev

            Python doticon star image 38689 doticonVersion:v2.0.2doticon License: Permissive (BSD-3-Clause)

            Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
            Support
              Quality
                Security
                  License
                    Reuse

                      numpyby numpy

                      Python doticonstar image 23755 doticonVersion:v1.25.0rc1doticon
                      License: Permissive (BSD-3-Clause)

                      The fundamental package for scientific computing with Python.

                      Support
                        Quality
                          Security
                            License
                              Reuse

                                numpyby numpy

                                Python doticon star image 23755 doticonVersion:v1.25.0rc1doticon License: Permissive (BSD-3-Clause)

                                The fundamental package for scientific computing with Python.
                                Support
                                  Quality
                                    Security
                                      License
                                        Reuse

                                          If you do not have pandas and numpy library that is required to run this code, you can install it by clicking on the above link.

                                          You can search for any dependent library on kandi like pandas and numpy.

                                          Support

                                          1. For any support on kandi solution kits, please use the chat
                                          2. For further learning resources, visit the Open Weaver Community learning page

                                          See similar Kits and Libraries

                                          Open Weaver – Develop Applications Faster with Open Source

                                          kandi

                                          Community and Support

                                          Company

                                          Follow

                                          • © 2023 Open Weaver Inc.