Introduction
Data is often hailed as the new oil in today's digital world, and Big Data has become an integral part of modern businesses and research. However, the quality of data used for analysis plays a pivotal role in deriving meaningful insights. Data cleaning, or data cleansing, is the process of identifying and correcting errors, inconsistencies, and inaccuracies in datasets.
In this Project, you will explore the fundamental principles of data cleaning in the context of Big Data, drawing from your in-depth understanding gained through various experiences, including labs, and applying these principles to a sample dataset.
Part 1: Research and Theory (1500 words) (12 marks)
Begin by researching and learning about the importance of data cleaning in the context of data analysis, statistics, and machine learning.
Explain why clean data is crucial for accurate analysis and decision-making.
Explain the different types of data and the 5Vs of Bigdata with examples
Describe common types of data quality issues, such as missing values, duplicates, outliers, and inconsistent formatting.
Explore and summarize at least three data cleaning techniques (e.g., handling missing data, removing duplicates, smoothing outliers) with explanations of how and when to use them.
Provide real-world examples or case studies where data cleaning played a vital role in achieving reliable results.
Part 2: Practical Application on Data Cleaning (15 marks)
Read the Sample Dataset that has been shared with you on Moodle.
Sample Dataset:
EmpID
Name
Age
Address
Profession
Salary
Marital status
301
Emma Reed
28
Sydney
Data Analyst
98000
Single
302
Liam Turner
41
Melbourne
UX Designer
112000
Married
303
Ava Mitchell
Nan
Brisbane
Nurse
67000
Divorced
304
Noah Patel
35
Nan
Project Manager
145000
Single
305
Mia Howard
29
Perth
Receptionist
52000
Married
Nan
Nan
Nan
Nan
Nan
Nan
Nan
306
Ethan Carter
55
Adelaide
CTO
980000
Married
307
Olivia Grant
32
Canberra
Data Scientist
129000
Single
303
Ava Mitchell
Nan
Brisbane
Nurse
67000
Divorced
308
Jacob Lee
27
Hobart
Intern
38000
Single
309
Sophia Baker
45
Darwin
HR Manager
118000
Married
310
Logan Perez
31
Gold Coast
Analyst
54000
Single
Your Job is to run your Pandas coding on the Dataset in Jupyter NoteBook and show every output including the code for the followings:
Show the first 5 rows of the Dataset
Explore the missing values in all the columns by indicating True (Null: yes) and False (Null: No).
Count the missing values in each column
Count non-missing values in each column
Drop missing values
Fill the missing values of all the String data (i.e. non-Numeric) in the Dataset with reasonable values (you may guess which values are suitable for which column)
Fill the missing values of all Numeric data of a column in the Dataset with its mean value.
Identify duplicate records in the Dataset.
Remove the duplicate records from the Dataset.
Find out which string data have whitespaces in it.
Address whitespaces from those strings
Change the string case (make all Capital) in your Dataset.
Explain the rationale behind each data cleaning step and how it contributes to improving the dataset's quality.
Part 3: Reflection and Conclusion (500 words) (1 mark)
Reflect on the overall data cleaning process. What did you learn from this experience?
Discuss the importance of documenting and maintaining data cleaning procedures for future reference.
Summarize the key takeaways from your research and practical application in a clear and concise manner.
Part 4: Group Contribution (500 words) (1 mark)
When assigning group projects, it's important to provide clear instructions on how you expect group members to contribute effectively. Here's a set of guidelines and instructions for group contributions in a group assignment:
Roles and Responsibilities: Clearly define the roles and responsibilities of each team member. Common roles include leader, researcher, writer, presenter, and editor. Emphasize that every member's role is vital to the project's success.
Communication: Encourage regular communication within the group through platforms like email, messaging apps, or project management tools. Provide contact information (emails, phone numbers) to facilitate easy communication.
Meetings: Specify how often and when group meetings should occur (e.g., weekly, bi-weekly). Emphasize the importance of attending meetings and suggest using video conferencing tools for virtual collaboration.
Contributions: Stress the significance of each member actively contributing to all aspects of the project (research, analysis, writing, presentation, etc.). Encourage brainstorming sessions and open discussions to facilitate idea sharing.
Encourage Collaboration: Encourage collaboration rather than just division of tasks. Emphasize the value of collective brainstorming, research, and problem-solving.
Project Progress: Require regular updates on the project's progress. Each member should report on their contributions during meetings or through written progress reports.
Consequences of Non-Participation: Clearly state the consequences of non-participation or lack of contribution. This may include individual grading penalties or removal from the group project.
Submission Guidelines:
Your essay should be typed in a standard font (e.g., Times New Roman, Arial) with 1.5 line spacing.
Include a cover page with your name, student ID, unit name & code, and assignment title.
Important Note: Plagiarism is strictly prohibited. Ensure that all sources are properly cited in your essay.
Timelines and Expectations:
Group Submission Due: 8th of Feb, 11:59pm
Expected word count: 2500 words.
Students are expected to submit the word file and Python codes via Turnitin on Moodle.
This assignment has a 5-day cutoff period. A 10% penalty per day (from the total marks) applies for each day the submission is late. Submission will not be accepted after 5-day cutoff period.
Marking Rubric:
Criteria
HD (High Distinction)
D (Distinction)
C (Credit)
P (Pass)
F (Fail)
Research and Theory (12 Marks)
Thorough and comprehensive explanation of data cleaning importance, types of data, 5Vs of Big Data, data quality issues, and cleaning techniques with real-world examples.
Detailed explanation with minor gaps in some areas.
Adequate explanation but lacks depth in some areas.
Basic explanation with significant gaps in understanding.
Incomplete or inaccurate explanation.
Practical Application on Data Cleaning (15 marks)
Accurate and thorough data cleaning process, all steps correctly implemented and well-explained, clear improvement of dataset quality.
Mostly accurate data cleaning process with minor errors, well-explained steps, noticeable improvement in dataset quality.
Adequate data cleaning process with some errors, steps explained but lacks depth, moderate improvement in dataset quality.
Basic data cleaning process with significant errors, poorly explained steps, limited improvement in dataset quality.
Incomplete or inaccurate data cleaning process, steps not well-explained, little to no improvement in dataset quality.
Reflection and Conclusion (1 Marks)
Clear, insightful reflection on the data cleaning process, importance of documentation well-discussed, concise summary of key takeaways.
Good reflection with minor insights, importance of documentation discussed, clear summary of key takeaways.
Adequate reflection with some insights, mentions documentation importance, moderate summary of key takeaways.
Basic reflection with limited insights, little mention of documentation, unclear summary of key takeaways.
Incomplete or inaccurate reflection, no mention of documentation, poor summary of key takeaways.
Group Contribution (1 Mark)
Clear roles and responsibilities defined, regular communication and collaboration, frequent updates on progress, consequences of non-participation well-discussed.
Mostly clear roles and responsibilities, regular communication, some updates on progress, consequences of non-participation discussed.
Adequate roles and responsibilities, some communication, occasional updates on progress, brief mention of consequences of non-participation.
Basic roles and responsibilities, limited communication, few updates on progress, little mention of consequences of non-participation.
Incomplete roles and responsibilities, poor communication, no updates on progress, no mention of consequences of non-participation.
Referencing (1 mark)
Referencing is thorough, accurate, and consistently follows the specified style guide without errors.
Referencing is complete and consistent with few errors.
Referencing is mostly complete and consistent with minor errors.
Referencing is present but incomplete or inconsistent.
Inadequate or no referencing provided.