API Reference
PrivateDataFrame
The PrivateDataFrame
API is based on pandas.DataFrame
, but in this case, all the methods are differentially private.
Constructor
The constructor for a PrivateDataFrame
is as follows:
Constructor:
class op_pandas.PrivateDataFrame(
df : pandas.DataFrame,
metadata = None,
categorical_metadata = None
)
Parameters:
df: pandas.DataFrame
: A pandas DataFrame, with data consisting of only strings, integers, floats, booleans, and datetime objects.metadata: Dict[str, Tuple(float,float)]
: Metadata containing bounds of the given DataFrame. The metadata should be a dictionary with column names as keys mapped to their bounds. Metadata contains keys of those columns that have only numerical data.{ 'Age': (18,65), 'Salary': (10000, 200000), 'Gender': (0,1) }
categorical_metadata: Dict[str, List]
: Metadata containing information about the categorical data of the given DataFrame. Thecategorical_metadata
should be a dictionary with column names as keys mapped to a list containing all the categories in the column. The data types for all the elements in the list must be identical.{ 'Income' : [">50k", "<=50k"], 'Sex': ["M", "F"] }
General Functions
applymap
The applymap()
method allows you to apply one or more functions to the DataFrame object, enabling the modification of each element independently.
Function Signature:
PrivateDataFrame.applymap(
func,
eps = 0,
output_bounds = None
) -> PrivateDataFrame:
Parameters:
func: callable
: Python function, returns a single value from a single value and should meet the following constraints:Func
can only take one argument, the individual element on which the function is applied.- Appropriate type annotations should be present in the function. To use datetime and regex,
import datetime
andimport re
to put their type annotations.
eps: float
: The epsilon provided to the differentially private calculation. Theeps
value must be>=0
. It’s used to calculate bounds.output_bounds: Dict[str, Tuple[float, float]]
: The output bounds (if already known) prevent the spending of epsilon from getting estimated bounds of the applied function.categorical_output_bounds: Dict[str, List]
: The categorical output bounds (if already known). If categorical output bounds for a specific column are not given, it will be calculated automatically using the function provided.
Returns:
PrivateDataFrame
: A new DataFrame with the function applied to each element.
all
The all()
method returns whether all elements are True, potentially over an axis.
Function Signature:
PrivateDataFrame.all(
axis: int = 0,
bool_only: bool = False,
skipna: bool = False
) -> PrivateSeries:
Parameters:
axis: int, default 0
: The axis to use. 0 is for rows, and 1 is for columns.bool_only: bool, default False
: Include only boolean columns. If False, all columns are included.skipna: bool, default False
: Exclude NA/null values when computing the result.
Returns:
PrivateSeries
: A Series indicating whether all elements along a specified axis are True.
categorical_metadata
This method returns the metadata of the categorical columns in PrivateDataFrame
.
Function Signature:
PrivateDataFrame.categorical_metadata -> dict
Returns:
dict
: A dictionary containing metadata about the categorical columns in the DataFrame.
columns
This method returns the column names of the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.columns -> list
Example:
>> priv_df.columns
['age', 'workclass', 'fnlwgt', 'education', 'educational-num',
'marital-status', 'occupation', 'relationship', 'race', 'gender',
'capital-gain', 'capital-loss', 'hours-per-week', 'native-country']
Returns:
list
: A list containing the names of the columns in the DataFrame.
copy
This method returns a copy of the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.copy() -> PrivateDataFrame
Returns:
PrivateDataFrame
: A new instance of PrivateDataFrame that is a copy of the original.
describe
The describe()
method returns a statistical description of the data in the DataFrame, using differentially private calculations.
Function Signature:
PrivateDataFrame.describe(
eps,
percentiles = None,
include = None,
exclude = None
)-> pandas.DataFrame
Parameters:
eps: float
: The epsilon provided to the differentially private calculation. eps must be>=0
.percentiles: list-like of numbers, optional
: The percentiles to include in the output. All should fall between 0 and 1. The default is [.25, .5, .75], which returns the 25th, 50th, and 75th percentiles.include: ‘all’, list-like of dtypes or None (default), optional
:all
: All input columns will be included in the output.- A list-like of dtypes: Limits the results to the provided data types.
- To limit the result to numeric types, submit
numpy.number
. - To limit the list to object columns submit the
numpy.object
data type. - Strings can also be used in the
select_dtypes
style. - To select pandas categorical columns, use
category
.
- To limit the result to numeric types, submit
- None (default): The result will include all numeric columns.
exclude: list-like of dtypes or None (default), optional
:- A list-like of dtypes : Excludes the provided data types from the result.
- To exclude numeric types submit
numpy.number
. - To exclude object columns submit the data type
numpy.object
. - Strings can also be used in the style of
select_dtype
(e.g.df.describe(exclude=['O'])
). - To exclude pandas’ categorical columns, use
category
.
- To exclude numeric types submit
- None (default): No result will be excluded.
- A list-like of dtypes : Excludes the provided data types from the result.
Returns:
pandas.DataFrame
: A DataFrame object with the statistical description of the DataFrame’s columns, adjusted for privacy concerns.
drop
The drop()
method removes the specified row or column from the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.drop(
columns=None,
inplace=True,
errors='raise'
)
Parameters:
columns: single label or list-like
: Alternative to specifying axis (labels, axis=1 is equivalent to columns=labels).inplace: boolean
: Whether to operate in place on the data.errors: {‘ignore’, ‘raise’}, default ‘raise’
: If you useignore
, suppress the error, and only existing labels are dropped.
dropna
The dropna()
method removes the rows that contain NULL values from the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.dropna(
axis=0,
how=_NoDefault.no_default,
thresh=_NoDefault.no_default
)
Parameters:
axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.how: str {‘any’, ‘all’}, default ‘any’
: Determine if a row or column is removed from DataFrame when we have at least one NA or all NA.any
: If any NA values are present, drop that row or column.all
: If all values are NA, drop that row or column.
thresh: int, optional
: Defines how many existing non-NA values are required to remove the row. It cannot be combined with how.
dtypes
The dtypes
property returns the data type of each column in the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.dtypes
fillna
The fillna()
method is used to replace missing values (NaN
s) in a PrivateDataFrame
. This method provides various options for filling missing data, either by specifying a static value or by using a method like 'forward fill' or 'backward fill'.
Function Signature:
PrivateDataFrame.fillna(
value=None,
limit: int = None,
method=None,
inplace: bool = False
):
Parameters:
value : scalar, dict, Series, DataFrame, or None, default None
: The value used to fill missing entries. It can be a scalar, dictionary, Series, or DataFrame, providing great flexibility in how replacements are handled. Ifvalue
is None andmethod
is specified, it will perform the specified method of filling.limit : int, optional
: The maximum number of consecutive NaN values to forward/backward fill. The limit applies to the number of filled values.method : {'backfill', 'bfill', 'pad', 'ffill', or None}, optional
: The method to use when filling holes in reindexed Series:'pad'
or'ffill'
: propagate last valid observation forward to next valid'backfill'
or'bfill'
: use NEXT valid observation to fill gap
inplace : bool, default False
: If True, fill in-place. Note that this will modify any other views on this object (e.g., a no-copy slice for a column in a DataFrame).
Returns:
PrivateDataFrame
or None: Depending on the value of inplace
, it either returns a new DataFrame with missing values filled or modifies the original DataFrame and returns None.
groupby
The groupby()
method on a PrivateDataFrame
is crucial for data analysis, allowing data to be grouped based on specific criteria and operations like sum, mean, and count to be executed on these groups.
Function Signature:
PrivateDataFrame.groupby(
by=None,
sort: bool=True,
dropna:
bool=True
) -> PrivateDataFrameGroupby
Parameters:
by : str | List | pd.Series | op_pandas.PrivateSeries
: Determines the groups for thegroupby
operation. Options include:- Column: Group by one or more categorical columns. The columns should be specified in the
categorical_metadata
. - Boolean Series / PrivateSeries: A series of boolean values. Non-boolean series will be converted to boolean before grouping.
- List: A combination of column names and series.
- Column: Group by one or more categorical columns. The columns should be specified in the
sort : bool, default True
: Controls whether the group keys are sorted. If set to False, the groups will appear in the order they are found in the original DataFrame.dropna : bool, default True
: If True, rows with NA values in the group keys are dropped. If False, NA values are included as a group key.
Allowed Operations:
After grouping, the following operations can be applied to compute statistics for each group:
- sum: Calculate the sum of group values.
- mean: Compute the average of group values.
- std: Standard deviation of the group values.
- var: Variance of the group values.
- count: Count of non-NA cells for each group.
- quantile: Compute quantiles for each group.
- median: Median of the group values.
- percentile: Specific percentiles of group values.
Returns:
PrivateDataFrameGroupby
: A specialized view of the DataFrame that supports further operations specific to groups.
Usage:
import op_pandas as opd
# Create a PrivateDataFrame with metadata
pdf = opd.PrivateDataFrame(
df,
metadata={"age": (0,100)},
categorical_metadata={"groups": ['a', 'b', 'c']}
)
# Group the data by 'groups' column
grouped = pdf.groupby("groups")
# Print the sum of the 'age' column for each group, ensuring differential privacy
print(grouped.sum(eps=1))
Output Example:
>>> age
a 837986.678085
b 817237.487139
c 827334.458893
This example demonstrates how to group data by categories and apply a privacy-preserving sum operation, providing insights into the dataset while maintaining the confidentiality of the data.
idxmax
The idxmax()
method returns the index of the first occurrence of the maximum value in the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.idxmax(axis=1, skipna=True, numeric_only=False) -> PrivateSeries:
Parameters:
axis: int, default 1
: The axis to use. Only column idxmax (axis=1) is allowed.skipna: bool, default True
: Exclude NA/null values when computing the result.numeric_only: bool, default False
: Include only float, int, and boolean columns.
idxmin
The idxmin()
method returns the index of the first occurrence of the minimum value in the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.idxmin(
axis=1,
skipna=True,
numeric_only=False
) -> PrivateSeries:
Parameters:
axis: int, default 1
: The axis to use. Only column idxmin (axis=1) is allowed.skipna: bool, default True
: Exclude NA/null values when computing the result.numeric_only: bool, default False
: Include only float, int, and boolean columns.
info
The info()
method provides a concise summary of a PrivateDataFrame
, detailing attributes like column names, their data types, and additional metadata concerning bounds and categorical distinctions.
Usage:
private_df.info()
isnull
The isnull()
method detects missing values for an array-like object.
Function Signature:
PrivateDataFrame.isnull() -> PrivateDataFrame:
isna
The isna()
method detects missing values for an array-like object.
Function Signature:
PrivateDataFrame.isna() -> PrivateDataFrame:
isin
The isin()
method checks if each element in a DataFrame is contained in the specified values.
Function Signature:
PrivateDataFrame.isin(values):
Parameters:
values: PrivateDataFrame
: The result will only be valid at a location if all the labels match.
join
The join()
method inserts columns from another DataFrame or Series.
Function Signature:
PrivateDataFrame.join(
other,
on = None,
how = "left",
lsuffix = "",
rsuffix = "",
sort = False,
validate = None
) -> PrivateDataFrame
Parameters:
other: PrivateDataFrame, PrivateSeries
: Index should be similar to one of the columns in this one. If a PrivateSeries is provided, its name attribute will be used as the column name in the resulting joined DataFrame.on: str, list of str, or array-like, optional
: Specifies in what level to do the joining.how{‘left’, ‘right’, ‘outer’, ‘inner’}, default ‘left’
:left
: use the calling frame’s index (or column if on is specified)right
: use the other’s index.outer
: form a union of the calling frame’s index (or column if one is specified) with the other’s index and sort it lexicographically.inner
: form the intersection of the calling frame’s index (or column if one is specified) with the other’s index, preserving the order of the calling’s one.
lsuffix: str, default ‘’
: Suffix to use from left frame’s overlapping columns.rsuffix: str, default ‘’
: Suffix to use from right frame’s overlapping columns.sort: bool, default False
: Order result DataFrame lexicographically by the join key. If False, the order of the join key depends on the join type (how keyword).validate: str, optional
: If specified, check if the join is of the specified type. The options are:one_to_one
or1:1
: Check if join keys are unique in both left and right datasets.one_to_many
or1:m
: Check if join keys are unique in the left dataset.many_to_one
orm:1
: Check if join keys are unique in the right dataset.many_to_many
orm:m
: This option is allowed but doesn’t result in checks.
make_column_categorical
The make_column_categorical
converts a noncategorical column to a categorical one.
Function Signature:
PrivateDataFrame.make_column_categorical(
column,
categories,
inplace=False
):
Parameters:
column: str
: Column to be converted to categorical.categories: List
: List of categories to be used for the column.inplace: bool
: If True, the operation is done in place.
make_column_non_categorical
The make_column_non_categorical
converts a categorical column to a noncategorical one.
Function Signature:
PrivateDataFrame.make_column_non_categorical(
columns: str | List[str],
output_bounds: dict = None,
eps: float = 0.0
)
Parameters:
columns: str | List[str]
: Column or a list of columns.output_bounds: dict
: If a column contains numerical values, but is categorical, you need to provide output bounds for it. If output bounds for a numerical column are absent, epsilon will be spent to estimate the bounds.eps: float
: Epsilon to estimate the output bounds of a numerical column.
metadata
The metadata
returns the metadata or bounds of numerical columns present in the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.metadata -> dict
notnull
The notnull()
method detects non-missing values for an array-like object.
Function Signature:
PrivateDataFrame.notnull() -> PrivateDataFrame:
notna
The notna()
method detects existing (non-missing) values.
Function Signature:
PrivateDataFrame.notna() -> PrivateDataFrame:
one_hot_encoding
The one_hot_encoding()
method encodes the categorical columns of the PrivateDataFrame
into one-hot vectors.
Function Signature:
PrivateDataFrame.one_hot_encoding(
cols,
prefix=None,
prefix_sep="_"
) -> PrivateDataFrame:
Parameters:
cols: str | List[str]
: Column or list of columns to be encoded.prefix: str
: Prefix to be used for the column names in the resultingPrivateDataFrame
.prefix_sep: str
: Separator to be used between the prefix and the column name.
rename
This method renames a specific set of columns in the PrivateDataFrame
. The rename
method uses a dictionary, which should contain a key-value pair of the one-to-one mapping needed for the column replacement.
PrivateDataFrame.rename(dict) -> PrivateDataFrame
sample_with_sensitivity
The sample_with_sensitivity()
method returns a random sample of items from the PrivateDataFrame
,
so that the sensitivity (how many times a user can be present in the dataset) is capped.
Function Signature:
PrivateDataFrame.sample_with_sensitivity(max_sensitivity) -> PrivateDataFrame:
Parameters:
max_sensitivity: int
: The maximum number of times a user can be present in the dataset.
size
The size
method returns the differentially private number of elements in the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.size(eps: float = 0) -> int:
Parameters:
eps: float
: The epsilon provided to the differentially private calculation. Theeps
value must be>=0
.
unique
The unique()
method returns the unique values in a column of the PrivateDataFrame
.
Function Signature:
PrivateDataFrame.unique(column: str) -> PrivateSeries:
Parameters:
column: str
: The column for which you want to find the unique values.
where
The where()
method replaces the values of the rows where the condition evaluates to False.
Function Signature:
PrivateDataFrame.where(
cond,
other = None,
inplace = False,
axis = None,
level = None
)
Parameters:
cond: bool PrivateSeries/PrivateDataFrame, Series/DataFrame, or array-like
:- If True, keep the original value.
- If False, replace it with the corresponding value from the other.
other: None
: Other tweaking is not supported currently.inplace: bool, default False
: Whether to operate in place on the data.axis: int, default None
: Alignment axis if needed. For Series, this parameter is unused and defaults to 0.level: int, default None
: Alignment level if needed.
Returns:
PrivateDataFrame
: A DataFrame with the result, or None if theinplace
parameter is set to True.
Basic statistical methods
count
The count()
method counts the number of not empty values for each row or column if you specify the axis parameter as axis='columns'
.
Function Signature:
PrivateDataFrame.count(
eps = 0,
axis=0,
numeric_only=False
)
Parameters:
eps : float, default = 0
: Inform the epsilon provided to the differentially private calculation. If axis = 0,eps
must be>=0
.axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.numeric_only: bool, default None
: Include only float, int, and boolean columns. If axis = 0,numeric_only
is always assumed to be True, otherwise, you must specify a value.
Returns:
Series
: A Series object with the count result for each row/column.
mean
The mean()
method returns the mean value of each column.
Function Signature:
PrivateDataFrame.mean(
eps = 0,
axis=0,
skipna=True,
numeric_only=None,
**kwargs
)
Parameters:
eps : float, default = 0
: Inform the epsilon provided to the differentially private calculation. If axis = 0,eps
must be>=0
.axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.skipna: bool, default True
: Exclude NA/null values when computing the result.numeric_only: bool, default None
: Include only float, int, and boolean columns. If axis = 0, numeric_only is always assumed to be True, otherwise, you must specify a value.*kwargs
: Additional keyword arguments are to be passed to the function.
Returns:
Series
: A Series with the mean values.
median
The median()
method returns a Series with the median value of each column.
Function Signature:
PrivateDataFrame.median(eps)
Parameters:
eps: float
: Inform the epsilon is provided for the differentially private calculation. Theeps
value must be>=0
.
percentile
It is a differentially private implementation of the percentile
method.
Function Signature:
PrivateDataFrame.percentile(eps, p)
Parameters:
eps: float
: Inform the epsilon provided to the differentially private calculation.eps
must be>=0
.p: float or array-like
: A value between0 <= p <= 100
. The percentile(s) to compute.
quantile
The quantile()
method calculates the quantile of the values in a given axis. The default axis is row.
Function Signature:
PrivateDataFrame.quantile(eps, q = 0.5)
Parameters:
eps: float
: Inform the epsilon provided to the differentially private calculation. Theeps
value must be>=0
.q: float or array-like, default 0.5 (50% quantile)
: A value between0 <= q <= 1
, the quantile(s) to compute.
Standard deviation
The standard deviation method, std()
, returns the sample’s standard deviation over a requested axis.
Function Signature:
PrivateDataFrame.std(
eps = 0,
axis=0,
skipna=True,
ddof=1,
numeric_only=None,
**kwargs
)
Parameters:
eps : float, default = 0
: Inform the epsilon provided to the differentially private calculation. Ifaxis = 0
,eps
must be>=0
.axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.skipna: bool, default True
: Exclude NA/null values when computing the result.ddof: int, default 1
: Delta Degrees of Freedom. The divisor used in calculations is , where N represents the number of elements. Ifaxis = 0
,ddof
must be equal to 1.numeric_only: bool, default None
: Include only float, int, boolean columns. Ifaxis = 0
,numeric_only
is always assumed to be True. Otherwise, you must specify a value.*kwargs
: The additional keyword arguments to be passed to the function.
sum
The sum()
method adds all values in each column and returns the sum for each one.
Function Signature:
PrivateDataFrame.sum(
eps = 0,
axis=0,
skipna=True,
numeric_only=None,
min_count=0,
**kwargs
)
Parameters:
eps: float, default = 0
: Inform the epsilon provided to the differentially private calculation. Ifaxis = 0
,eps
must be>=0
.axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.skipna: bool, default True
: Exclude NA/Null values when computing the result.numeric_only: bool, default None
: Include only float, int, and boolean columns. Ifaxis = 0
,numeric_only
is always assumed to be True. Otherwise, you must specify a value.min_count: int, default 0
: The required number of valid values to operate.- If fewer than
min_count
non-NA values are present, the result will be NA. - If
axis = 0
,min_count
is always assumed to be 0. Otherwise, you must specify a value.
- If fewer than
*kwargs
: Additional keyword arguments to be passed to the function.
variance
The var()
method calculates the variance for each column.
Function Signature:
PrivateDataFrame.var(eps = 0, axis=0, skipna=True, ddof=1, numeric_only=None, **kwargs)
Parameters:
eps : float, default = 0
: Inform the epsilon provided to the differentially private calculation. Ifaxis = 0
,eps
must be>=0
.axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.skipna: bool, default True
: Exclude NA/Null values when computing the result.ddof: int, default 1
: Delta Degrees of Freedom. The divisor used in calculations is , where N represents the number of elements. Ifaxis = 0
,ddof
must be equal to 1.numeric_only: bool, default None
: Include only float, int, and boolean columns. Ifaxis = 0
,numeric_only
is always assumed to be True. Otherwise, you must specify a value.*kwargs
: Additional keyword arguments are to be passed to the function.
Advanced statistical methods
correlation
The corr()
method finds the correlation of each column in a PrivateDataFrame
.
Function Signature:
PrivateDataFrame.corr(eps: float, method: str = "pearson", min_periods: int = 1, numeric_only = True)
Parameters:
eps: float
: Inform the epsilon provided to the differentially private calculation. Theeps
value must be>=0
.method: str, {‘pearson’ or ‘spearman’}, default 'pearson'
: Define the method used to calculate the correlation. The available options are:- pearson : standard correlation coefficient.
- spearman : Spearman rank correlation.
min_periods: int, optional
: Assumed to be 1. Currently,min_periods
tweaking is not supported.numeric_only: bool, default True
: Include only float, int, or boolean data. Currently,numeric_only
tweaking is not allowed.
Returns:
Pandas DataFrame
: A DataFrame object with the correlation results.
covariance
The cov()
method finds the covariance of each column in a PrivateDataFrame
.
Function Signature:
PrivateDataFrame.cov(
eps: float,
min_periods,
ddof = 1,
numeric_only = True
)
Parameters:
eps: float
: Inform the epsilon provided to the differentially private calculation. Theeps
value must be>=0
.min_periods: int, optional
: Assumed to be 1. Currently,min_periods
tweaking is not supported.ddof: int, default 1
: Delta Degrees of Freedom. The divisor used in calculations is , where N represents the number of elements. Currently,ddof
tweaking is not supported.numeric_only: bool, default True
: Include only float, int, or boolean data. Currently,numeric_only
tweaking is not allowed.
Returns:
Pandas DataFrame
: A DataFrame object with the covariance results.
skew
The skew()
method calculates the skew for each column.
Function Signature:
PrivateDataFrame.skew(
eps,
axis = 0,
skipna = True,
numeric_only = True
)
Parameters:
eps : float, default = 0
: Inform the epsilon provided to the differentially private calculation. If
axis = 0
,eps
must be>=0
.axis: boolean {index (0), columns (1)}, default = 0
: Axis for the function to be applied on.skipna: bool, default True
: Exclude NA/Null values when computing the result.numeric_only: bool, default True
: Include only float, int, and boolean columns. Ifaxis = 0
,numeric_only
is always assumed to be True. Otherwise, you must specify a value.
Returns:
Pandas DataFrame
: A DataFrame object with the skew results.
Histograms
hist
This method draws a histogram of the PrivateDataFrame’s columns.
Function Signature:
PrivateDataFrame.hist(
column,
eps,
bins = 10
)
Parameters:
column: str
: Column in thePrivateDataFrame
to group by.eps: float
: Inform the epsilon provided to the differentially private calculation. Theeps
value must be>=0
.bins: int, default 10
: Number of histogram bins to be used.
hist2d
This method creates a 2D histogram among two of the columns of the PrivateDataFrame.
Function Signature:
PrivateDataFrame.hist2d(eps, x, y, bins = 10):
Parameters:
eps: float
: The epsilon provided to the differentially private calculation. Theeps
value must be>=0
.x: str
: Inform the first column to be used from thePrivateDataFrame
to group by.y: str
: Inform the second column to be used from thePrivateDataFrame
to group by.bins: int, default 10
: Number of histogram bins to be used.