In this tutorial series we build a Chatbot with TensorFlow's sequence to sequence library and by building a massive database from Reddit comments.
Text tutorials and sample code:
Monthly compressed comment dumps:
1.7bln comments:
ONE MONTH magnet: magnet:?xt=urn:btih:32916ad30ce4c90ee4c47a95bd0075e44ac15dd2&dn=RC%5F2015-01.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969
FULL 1.7bln comment torrent:
magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&dn=reddit%5Fdata&tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80
Loaded on bigquery:
And:
Your tutorials are excellent!!!
Thank you for starting this series. Looking forward to follow along this one!
The wait has been over!!! Love the way you teach!
Nested queries are more time effective. Exemple : WHERE name=name AND address=address is costing a lot more because it is searching all the database each time to compare. However, the nested query WHERE name=name AND ( SELECT address …) is costing less because it has pre-selected a portion of the database to compare. Hope this helps!
Great intro, what minimum python skill do I need to get building/programming my own chatbot? Thank you for so many wonderful series/tutorials!
You make such an interesting projects, I wish I had time (and skills) to make them too!
this is an exciting tutorial. I’d love to see the results of the full data set.
That’d be a LOT of data. Maybe one day.
So turns out you can run a parameterized BQ Query and state your parameters in a JSON file. This gives you the flexibility to change different parameters like YEAR and the like without rewriting the query here goes and example using the New York taxi dataset in BQ.
Step 1 Define New Query
%bq query -n taxiquery
WITH trips AS (
SELECT EXTRACT (DAYOFYEAR from pickup_datetime) AS day number FROM ‘nyc-tlc.green.trips_*
WHERE _TABLE_SUFFIX = @YEAR
)
SELECT daynumber, COUNT(I) AS numtrips from trips
GROUP BY daynumber ORDER BY daynumber
Step 2 Setup JSON Query Parameter for @YEAR
query_parameter = [
{
‘name’ : = ‘YEAR’,
‘parameterType’ : {‘type’ : ‘STRING’ },
‘parameterValue’: {‘value’: 2015}
}
]
Step 3: Execute the query
trips = taxiquery.execute(query_params=query_parameter).result().to_dataframe()
Explore top 5 Records
trips[5]
BQ Charges by amount of Data in Query and the amount of Data in results. By saving the results to a data frame you’ll save money from future queries.
SELECT * queries are the most expensive.
sentedex I love your videos especially that you are a python programmer I love python can you show us how can we make an IP scanner and how to connect to it [with python] please 🙂
Bro you are simply awesome !!!super learner Wish I could learn more from you!!
This is the first video of the series and I already got a feeling this is going to be epic !! Thanks a lot for your efforts Harrison!
I am inspired to study python because of your tutorials. So detailed and fun to watch. Jeffrey Way of python (:
A chatbot could be worked with over time to eventually be developed into a peronal assistant right? So, like after the chat/conversation aspect is taken care of you could work in things like simple tasks? Like opening an app/software for example. And what if you wanted to add in a way of voice recognition? Is that something simply added to the chatbot or would it require essentialy a ton of rewriting? Sorry for all the questions. I’ve been learning python but have just gotten into the whole machine learning/chatbot/ AI aspect. With the Virtual Assitent aspect as my goal I was wondering if I should be working with your machine learning series instead of the chatbot one. You seem to know a lot and I’ve already learned quite a bit from watching you so far so thanks for the videos and keep up the good work. 🙂
Hey sentdex, thanks for this amazing tutorial. I’m new to deep learning and I have a question.
How long (approximately) will it take this model to train on my laptop with these specs – 2GB GPU, Core i3 processor, 6 GB RAM, win 10 64 bit.
Love your work! Thanks for the videos!
You are literally my favorite youtuber, thank you for making the best python content ever!
Hey, I would love your advice on how do I start building chatbots? Do I simply follow your video series or how do I create something of my own from scratch?
I found your videos from your golang series. Just started looking into AI more and stumbled across this series. Thanks for creating it. I am really excited to see where it takes me.
Hey sentdex! I love your videos very much and i am a subscriber of yours! Can you make a chatbot tutorial with AIML in python? Please!
For Ubuntu users, best practice to extract bz2 file is by:
bzip2 -dk RC_2015-05.bz2
Hope it helps!