• Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    AI

    Fraud detection beyond a single transaction: timing, account links and data quality

    Drawing on their time at Airbnb, a user describes combining real-time and batch data to detect fraud—and warns that good overall evaluation results can hide regressions in specific segments.

    2 Sources, 20d ago, first seen 20d ago

    TLDR

    A user sharing lessons from their time at Airbnb outlines four kinds of fraud signals: details of the event, the account’s state, the sequence and speed of activity, and connections between accounts. They describe an early system that extracted features, trained models on labeled outcomes and produced fraud-risk scores.

    Their advice: shorten the evaluation loop, inspect results by segment, and check offline tests for duplicates, overlap between training and test data, and features that accidentally include future information. They also recommend keeping raw and converted currency amounts, timestamping external data such as exchange rates, and investing in clean data and reliable labels.

    Combined views

    —

    2 Sources, first seen 20d ago

    — likes— comments— saves— reposts

    Combined views

    —

    2 Sources, first seen 20d ago

    — likes— comments— saves— reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Featured Source

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    2 Sources

    Dmitry Alexeenko@dalexeenko1/ Fraud and safety topics seem to be as important as ever. Here are some notes / lessons learned on fraud detection from my time at Airbnb. Let's start with a simple [made-up] transaction: IP location: 213.30.123.456 (Lisbon) Billing address: San Francisco, CA Name: Dom Cobb Email: ariadne@u.washington.edu Credit card: 4242 4242 4242 4242 Auth result: Success Good questions to ask are: Is user IP consistent with their billing address? What about their name and their email address? What's the reputation of their email provider? These are your features. This won't be enough, but it's a good start. 2/ Now add the time dimension. You want to look at both sequence and velocity: Account created → card added → purchase → card changed → two more purchases If an account is 4 hours old and they had 2 credit card updates and 3 transactions in the last hour, that might be worth looking into. This is where RNNs and LSTMs shine. What was the sequence and how quickly did the user go through that sequence? 3/ Now zoom out across accounts and look at the graph. Each individual account might look normal on its own. What matters is how these accounts are connected and what properties or behaviors they share. That can tell you something you can't see in a single account. 4/ An early version of our system had 3 main pieces: (1) Feature extraction, that turned raw events into features, (2) Model training, that learned from labeled outcomes, (3) Scoring, that combined feature vectors with trained models and produced a P("Bad") score. 5/ Features will come from widely different sources. The difficult part is to efficiently retrieve 100s of complex features that rely on data across services, databases, APIs, vendors. We used a Lambda architecture to combine real-time streaming data with batch computations, bringing these together through a highly-efficient feature store. 6/ A useful way to categorize features: - Event: what just happened? E.g., credit card type, billing address, property of the item purchased. - State: what do we know about this account? E.g., country, browser fingerprint, cookie, etc. - Temporal: what happened before, and how quickly? Both sequences, like the last 5 geolocations, and velocity, like the number of purchases in the last hour. - Graph: how is it connected to other accounts? E.g., number of users using the same credit card. 7/ You need to have a way to assess whether the feature is useful. E.g., does this new feature add anything the model doesn't already know? That's wher you spend time on information gain / KL divergence. E.g., you can find out that billing address capitalization has a KL of 0.006, which may not be that useful. While the shipping/billing country match may have a KL of 0.35. That's much better! 8/ Some lessons learned: - The speed of eval loop is critical. Idea → code → dataset → features → train → evaluate → deploy → monitor. The tighter you can make the loop, the better. - Look at evals overall, but also dig inside the averages. Where did the ROC curve improve? What happens at the threshold you use? It's critical to break results down by segment. Often times you'll find local regressions hiding behind nice aggregate numbers. - Be careful looking at offline results. Check for duplicate examples, train/test overlap, features that accidentally include information about the future. - Units of measure matter. A transaction amount of 1,000 means different things in USD vs JPY. You want both raw and converted amounts. - External knowledge needs to have a timestamp. Drawing from the previous example, FX rates change overtime. Geo IP databases change. Card metadata changes. - Finally: noise is everywhere. You'll have to deal with wrong labels, duplicates, missing fields, stale data, etc. You can then spend weeks tuning a model around messy data. So invest in obtaining and cleaning ground truth.20d
    Jeff Weinstein@jeff_weinsteinRT @dalexeenko: 1/ Fraud and safety topics seem to be as important as ever. Here are some notes / lessons learned on fraud detection from m…20d
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    2 Sources

    Dmitry Alexeenko@dalexeenko1/ Fraud and safety topics seem to be as important as ever. Here are some notes / lessons learned on fraud detection from my time at Airbnb. Let's start with a simple [made-up] transaction: IP location: 213.30.123.456 (Lisbon) Billing address: San Francisco, CA Name: Dom Cobb Email: ariadne@u.washington.edu Credit card: 4242 4242 4242 4242 Auth result: Success Good questions to ask are: Is user IP consistent with their billing address? What about their name and their email address? What's the reputation of their email provider? These are your features. This won't be enough, but it's a good start. 2/ Now add the time dimension. You want to look at both sequence and velocity: Account created → card added → purchase → card changed → two more purchases If an account is 4 hours old and they had 2 credit card updates and 3 transactions in the last hour, that might be worth looking into. This is where RNNs and LSTMs shine. What was the sequence and how quickly did the user go through that sequence? 3/ Now zoom out across accounts and look at the graph. Each individual account might look normal on its own. What matters is how these accounts are connected and what properties or behaviors they share. That can tell you something you can't see in a single account. 4/ An early version of our system had 3 main pieces: (1) Feature extraction, that turned raw events into features, (2) Model training, that learned from labeled outcomes, (3) Scoring, that combined feature vectors with trained models and produced a P("Bad") score. 5/ Features will come from widely different sources. The difficult part is to efficiently retrieve 100s of complex features that rely on data across services, databases, APIs, vendors. We used a Lambda architecture to combine real-time streaming data with batch computations, bringing these together through a highly-efficient feature store. 6/ A useful way to categorize features: - Event: what just happened? E.g., credit card type, billing address, property of the item purchased. - State: what do we know about this account? E.g., country, browser fingerprint, cookie, etc. - Temporal: what happened before, and how quickly? Both sequences, like the last 5 geolocations, and velocity, like the number of purchases in the last hour. - Graph: how is it connected to other accounts? E.g., number of users using the same credit card. 7/ You need to have a way to assess whether the feature is useful. E.g., does this new feature add anything the model doesn't already know? That's wher you spend time on information gain / KL divergence. E.g., you can find out that billing address capitalization has a KL of 0.006, which may not be that useful. While the shipping/billing country match may have a KL of 0.35. That's much better! 8/ Some lessons learned: - The speed of eval loop is critical. Idea → code → dataset → features → train → evaluate → deploy → monitor. The tighter you can make the loop, the better. - Look at evals overall, but also dig inside the averages. Where did the ROC curve improve? What happens at the threshold you use? It's critical to break results down by segment. Often times you'll find local regressions hiding behind nice aggregate numbers. - Be careful looking at offline results. Check for duplicate examples, train/test overlap, features that accidentally include information about the future. - Units of measure matter. A transaction amount of 1,000 means different things in USD vs JPY. You want both raw and converted amounts. - External knowledge needs to have a timestamp. Drawing from the previous example, FX rates change overtime. Geo IP databases change. Card metadata changes. - Finally: noise is everywhere. You'll have to deal with wrong labels, duplicates, missing fields, stale data, etc. You can then spend weeks tuning a model around messy data. So invest in obtaining and cleaning ground truth.20d
    Jeff Weinstein@jeff_weinsteinRT @dalexeenko: 1/ Fraud and safety topics seem to be as important as ever. Here are some notes / lessons learned on fraud detection from m…20d
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet