How to use 'pandas drop duplicates' in Python

Every line of 'pandas drop duplicates' code snippets is scanned for vulnerabilities by our powerful machine learning engine that combs millions of open source libraries, ensuring your Python code is secure.

All examples are scanned by Snyk Code

By copying the Snyk Code Snippets you agree to
this disclaimer
49def remove_duplicates(df_or_series):
50 """ Remove duplicate rows or values by keeping the first of each duplicate.
51
52 Parameters
53 ----------
54 df_or_series : :any:`pandas.DataFrame` or :any:`pandas.Series`
55 Pandas object from which to drop duplicate index values.
56
57 Returns
58 -------
59 deduplicated : :any:`pandas.DataFrame` or :any:`pandas.Series`
60 The deduplicated pandas object.
61 """
62 # CalTrack 2.3.2.2
63 return df_or_series[~df_or_series.index.duplicated(keep="first")]
Important

Use secure code every time

Secure your code as it's written. Use Snyk Code to scan source code in minutes – no build needed – and fix issues immediately. Enable Snyk Code

14def drop_duplicate_events(df):
15 """
16 Function to group dataframe, use all new information from the latest row
17 but keep the ``event_index`` from the first one
18 """
19 df = df.sort_values('event_index', na_position='last')
20 event_index = df.event_index.iloc[0]
21 r = df.iloc[-1].to_dict()
22 r['event_index'] = event_index
23 return r

Related snippets