cdsc_reddit.git
2 months agoMerge branch 'master' of code:cdsc_reddit master
Nathan TeBlunthuis [Tue, 3 Aug 2021 22:03:40 +0000 (15:03 -0700)]
Merge branch 'master' of code:cdsc_reddit

2 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Wed, 28 Jul 2021 22:32:21 +0000 (15:32 -0700)]
Merge branch 'master' of code:cdsc_reddit

2 months agono longer do we need to get daily dumps
Nate E TeBlunthuis [Wed, 28 Jul 2021 22:32:04 +0000 (15:32 -0700)]
no longer do we need to get daily dumps

5 months agobugfix
Nate E TeBlunthuis [Tue, 27 Apr 2021 05:31:05 +0000 (22:31 -0700)]
bugfix

5 months agoMerge branch 'charliepatch' of code:cdsc_reddit into charliepatch
Nate E TeBlunthuis [Mon, 26 Apr 2021 20:22:29 +0000 (13:22 -0700)]
Merge branch 'charliepatch' of code:cdsc_reddit into charliepatch

5 months agosupport passing in list of tfidf vectors.
Nate E TeBlunthuis [Mon, 26 Apr 2021 18:16:28 +0000 (11:16 -0700)]
support passing in list of tfidf vectors.

Also lowercases included subreddits.

5 months agosupport passing in list of tfidf vectors.
Nate E TeBlunthuis [Mon, 26 Apr 2021 18:16:28 +0000 (11:16 -0700)]
support passing in list of tfidf vectors.

Also lowercases included subreddits.

6 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Thu, 22 Apr 2021 17:46:26 +0000 (10:46 -0700)]
Merge branch 'master' of code:cdsc_reddit

6 months agoversion of weekly_cosine_similarities.py from klone
Nate E TeBlunthuis [Thu, 22 Apr 2021 17:38:10 +0000 (10:38 -0700)]
version of weekly_cosine_similarities.py from klone

6 months agobugfix in weekly similarities
Nate E TeBlunthuis [Thu, 22 Apr 2021 17:37:04 +0000 (10:37 -0700)]
bugfix in weekly similarities

6 months agobugfixes in clustering selection.
Nate E TeBlunthuis [Wed, 21 Apr 2021 23:56:25 +0000 (16:56 -0700)]
bugfixes in clustering selection.

6 months agocalculate some user-level attributes to detect bots
Nate E TeBlunthuis [Tue, 20 Apr 2021 18:34:36 +0000 (11:34 -0700)]
calculate some user-level attributes to detect bots

6 months agogrid sweep selection for clustering hyperparameters
Nate E TeBlunthuis [Tue, 20 Apr 2021 18:33:54 +0000 (11:33 -0700)]
grid sweep selection for clustering hyperparameters

6 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Tue, 6 Apr 2021 06:21:35 +0000 (23:21 -0700)]
Merge branch 'master' of code:cdsc_reddit

6 months agoChanges for cosine similarities on klone.
Nate E TeBlunthuis [Tue, 6 Apr 2021 06:21:06 +0000 (23:21 -0700)]
Changes for cosine similarities on klone.

7 months agoadd code for pulling activity time series from parquet.
Nate E TeBlunthuis [Wed, 24 Mar 2021 23:08:57 +0000 (16:08 -0700)]
add code for pulling activity time series from parquet.

8 months agoadd included_subreddits parameter to cosine similarities.
Nate E TeBlunthuis [Tue, 23 Feb 2021 02:38:34 +0000 (18:38 -0800)]
add included_subreddits parameter to cosine similarities.

8 months agoChanges from hyak.
Nate E TeBlunthuis [Tue, 23 Feb 2021 00:03:48 +0000 (16:03 -0800)]
Changes from hyak.

8 months agofix bug in viz.
Nate E TeBlunthuis [Thu, 28 Jan 2021 04:26:15 +0000 (20:26 -0800)]
fix bug in viz.

8 months agoadd visualization for 10000 subreddits based on author-tf similarities.
Nate E TeBlunthuis [Thu, 28 Jan 2021 04:22:24 +0000 (20:22 -0800)]
add visualization for 10000 subreddits based on author-tf similarities.

8 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Thu, 28 Jan 2021 04:09:23 +0000 (20:09 -0800)]
Merge branch 'master' of code:cdsc_reddit

8 months agoadd cluster selection to visualization
Nathan TeBlunthuis [Thu, 28 Jan 2021 04:08:07 +0000 (20:08 -0800)]
add cluster selection to visualization

9 months agoremove nsfw subs from topN
Nate E TeBlunthuis [Tue, 29 Dec 2020 05:11:44 +0000 (21:11 -0800)]
remove nsfw subs from topN

10 months agoUpdating to support wang-style user overlaps.
Nate E TeBlunthuis [Fri, 25 Dec 2020 06:38:04 +0000 (22:38 -0800)]
Updating to support wang-style user overlaps.

10 months agoSome improvements to run affinity clustering on larger dataset and
Nate E TeBlunthuis [Sun, 13 Dec 2020 04:42:47 +0000 (20:42 -0800)]
Some improvements to run affinity clustering on larger dataset and
compute density.

10 months agoRefactor and reorganze.
Nate E TeBlunthuis [Wed, 9 Dec 2020 01:32:20 +0000 (17:32 -0800)]
Refactor and reorganze.

10 months agoAdd code for running tf-idf at the weekly level.
Nate E TeBlunthuis [Wed, 2 Dec 2020 06:54:48 +0000 (22:54 -0800)]
Add code for running tf-idf at the weekly level.

11 months agorefactor visualization code.
Nathan TeBlunthuis [Wed, 18 Nov 2020 00:46:49 +0000 (16:46 -0800)]
refactor visualization code.

11 months agoMerge remote-tracking branch 'refs/remotes/origin/master' into master synced/master
Nathan TeBlunthuis [Wed, 18 Nov 2020 00:33:14 +0000 (16:33 -0800)]
Merge remote-tracking branch 'refs/remotes/origin/master' into master

11 months agogit-annex in nathante@nate-x1:~/cdsc_reddit
Nathan TeBlunthuis [Wed, 18 Nov 2020 00:33:13 +0000 (16:33 -0800)]
git-annex in nathante@nate-x1:~/cdsc_reddit

11 months agogit-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit
Nate E TeBlunthuis [Wed, 18 Nov 2020 00:31:48 +0000 (16:31 -0800)]
git-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit

11 months agoUpdate code for clustering + tsne.
Nate E TeBlunthuis [Tue, 17 Nov 2020 23:59:20 +0000 (15:59 -0800)]
Update code for clustering + tsne.

11 months agoUpdate code for building simlarity matrices.
Nate E TeBlunthuis [Tue, 17 Nov 2020 20:52:48 +0000 (12:52 -0800)]
Update code for building simlarity matrices.

11 months agobugfix in completing tfidf similarity matrices.
Nate E TeBlunthuis [Thu, 12 Nov 2020 19:47:53 +0000 (11:47 -0800)]
bugfix in completing tfidf similarity matrices.

11 months agoincrease learning rate.
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:58:39 +0000 (16:58 -0800)]
increase learning rate.

11 months agoincrease iterations and perplectity and early_exaggeration
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:55:39 +0000 (16:55 -0800)]
increase iterations and perplectity and early_exaggeration

11 months agoincrease learning rate
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:48:41 +0000 (16:48 -0800)]
increase learning rate

11 months agoFix bug in tsne.
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:43:41 +0000 (16:43 -0800)]
Fix bug in tsne.

11 months agogit-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:39:44 +0000 (16:39 -0800)]
git-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit

11 months agosplit fitting and plotting tsne.
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:38:22 +0000 (16:38 -0800)]
split fitting and plotting tsne.

11 months agoAdd file to plot related subreddits using tsne.
Nathan TeBlunthuis [Thu, 12 Nov 2020 00:05:36 +0000 (16:05 -0800)]
Add file to plot related subreddits using tsne.

11 months agoBugfix (typo)
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:38:11 +0000 (13:38 -0800)]
Bugfix (typo)

11 months agoReuse code for term and author cosine similarity.
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:18:57 +0000 (13:18 -0800)]
Reuse code for term and author cosine similarity.

11 months agoRefactor tfidf code to for code resuse.
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:18:19 +0000 (13:18 -0800)]
Refactor tfidf code to for code resuse.

11 months agorename 'idf' files to 'tfidf'
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:16:55 +0000 (13:16 -0800)]
rename 'idf' files to 'tfidf'

11 months agoImprovements to idf code
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:12:11 +0000 (13:12 -0800)]
Improvements to idf code

11 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Mon, 2 Nov 2020 18:40:12 +0000 (10:40 -0800)]
Merge branch 'master' of code:cdsc_reddit

11 months agoadd term_cosine_similarity.py
Nate E TeBlunthuis [Mon, 2 Nov 2020 18:40:02 +0000 (10:40 -0800)]
add term_cosine_similarity.py

11 months agoAdd Cosine similarities to README.md
Nathan TeBlunthuis [Mon, 2 Nov 2020 17:48:10 +0000 (09:48 -0800)]
Add Cosine similarities to README.md

11 months agoUpdate Readme.
Nathan TeBlunthuis [Mon, 2 Nov 2020 16:42:13 +0000 (08:42 -0800)]
Update Readme.

11 months agoMerge branch 'master' of code:cdsc_reddit into master
Nathan TeBlunthuis [Mon, 2 Nov 2020 05:50:44 +0000 (21:50 -0800)]
Merge branch 'master' of code:cdsc_reddit into master

11 months agoCreate README.md
Nathan TeBlunthuis [Mon, 2 Nov 2020 05:50:27 +0000 (21:50 -0800)]
Create README.md

12 months agoUpdate reddit comments data with daily dumps.
Nate E TeBlunthuis [Sat, 3 Oct 2020 23:42:22 +0000 (16:42 -0700)]
Update reddit comments data with daily dumps.

14 months agoCompute IDF for terms and authors.
Nate E TeBlunthuis [Sun, 23 Aug 2020 18:57:55 +0000 (11:57 -0700)]
Compute IDF for terms and authors.

14 months agoUpdate submissions to parse using the backfill queue.
Nate E TeBlunthuis [Wed, 12 Aug 2020 05:37:36 +0000 (22:37 -0700)]
Update submissions to parse using the backfill queue.

14 months agobugfix in checking submission shas
Nate E TeBlunthuis [Tue, 11 Aug 2020 21:21:54 +0000 (14:21 -0700)]
bugfix in checking submission shas

14 months agoUse multiword expressions in tf.
Nate E TeBlunthuis [Mon, 10 Aug 2020 23:57:46 +0000 (16:57 -0700)]
Use multiword expressions in tf.

14 months agoFinish generating multiword expressions.
Nate E TeBlunthuis [Mon, 10 Aug 2020 05:42:23 +0000 (22:42 -0700)]
Finish generating multiword expressions.

14 months agoBugfix
Nate E TeBlunthuis [Sun, 9 Aug 2020 09:34:42 +0000 (02:34 -0700)]
Bugfix

14 months agoUse groupby - joins instead of windows
Nate E TeBlunthuis [Sun, 9 Aug 2020 07:21:50 +0000 (00:21 -0700)]
Use groupby - joins instead of windows

14 months agorenamte tf_comments part 2.
Nate E TeBlunthuis [Tue, 4 Aug 2020 20:39:49 +0000 (13:39 -0700)]
renamte tf_comments part 2.

14 months agorename tf_reddit_comments.py step1.
Nate E TeBlunthuis [Tue, 4 Aug 2020 20:39:20 +0000 (13:39 -0700)]
rename tf_reddit_comments.py step1.

14 months agoImprove tokenization following data. Generate author counts.
Nate E TeBlunthuis [Tue, 4 Aug 2020 20:24:37 +0000 (13:24 -0700)]
Improve tokenization following data. Generate author counts.

14 months agoimprove tokenizer.
Nate E TeBlunthuis [Tue, 4 Aug 2020 05:55:10 +0000 (22:55 -0700)]
improve tokenizer.

14 months agoTF reddit comments.
Nate E TeBlunthuis [Tue, 4 Aug 2020 05:43:57 +0000 (22:43 -0700)]
TF reddit comments.

14 months agocode to sort tf
Nate E TeBlunthuis [Tue, 4 Aug 2020 00:56:36 +0000 (17:56 -0700)]
code to sort tf

15 months agoremove is_submitter field from submissions which doesn't exist.
Nate E TeBlunthuis [Fri, 10 Jul 2020 00:12:14 +0000 (17:12 -0700)]
remove is_submitter field from submissions which doesn't exist.

15 months agoBugfixes in scripts.
Nate E TeBlunthuis [Wed, 8 Jul 2020 06:29:36 +0000 (23:29 -0700)]
Bugfixes in scripts.

15 months agoclean up comments in streaming example.
Nate E TeBlunthuis [Tue, 7 Jul 2020 19:28:57 +0000 (12:28 -0700)]
clean up comments in streaming example.

15 months agoupdate .gitignore
Nate E TeBlunthuis [Tue, 7 Jul 2020 19:28:44 +0000 (12:28 -0700)]
update .gitignore

15 months agoupdate examples with working streaming
Nate E TeBlunthuis [Tue, 7 Jul 2020 18:47:17 +0000 (11:47 -0700)]
update examples with working streaming

15 months agoBuild comments dataset similarly to submissions and improve partitioning scheme
Nate E TeBlunthuis [Tue, 7 Jul 2020 18:45:43 +0000 (11:45 -0700)]
Build comments dataset similarly to submissions and improve partitioning scheme

15 months agoupdate .gitignore
Nate E TeBlunthuis [Tue, 7 Jul 2020 07:58:26 +0000 (00:58 -0700)]
update .gitignore

15 months agoScript for example of streaming pyarrow.
Nate E TeBlunthuis [Tue, 7 Jul 2020 07:57:05 +0000 (00:57 -0700)]
Script for example of streaming pyarrow.

15 months agoScript to demonstrate reading parquet.
Nate E TeBlunthuis [Tue, 7 Jul 2020 07:51:40 +0000 (00:51 -0700)]
Script to demonstrate reading parquet.

15 months agoCheck the shas when we download dumps
Nate E TeBlunthuis [Tue, 7 Jul 2020 06:31:52 +0000 (23:31 -0700)]
Check the shas when we download dumps

15 months agoScript to run both parts of submissions_2_parquet.sh
Nate E TeBlunthuis [Tue, 7 Jul 2020 06:27:18 +0000 (23:27 -0700)]
Script to run both parts of submissions_2_parquet.sh

15 months agoCache before sorting so we don't extract twice.
Nate E TeBlunthuis [Tue, 7 Jul 2020 05:30:04 +0000 (22:30 -0700)]
Cache before sorting so we don't extract twice.

15 months agoMove the spark part of submissions_2_parquet to a separate script.
Nate E TeBlunthuis [Tue, 7 Jul 2020 05:26:29 +0000 (22:26 -0700)]
Move the spark part of submissions_2_parquet to a separate script.

15 months agoFix whitespace at top of file.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:32:00 +0000 (23:32 -0700)]
Fix whitespace at top of file.

15 months agoSecondary sort for the by_author dataset should be CreatedAt.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:27:18 +0000 (23:27 -0700)]
Secondary sort for the by_author dataset should be CreatedAt.

15 months agoCreate a second dataset sorted by author.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:24:40 +0000 (23:24 -0700)]
Create a second dataset sorted by author.

15 months agoCreate parquet datasets of reddit submissions from pushshift.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:20:17 +0000 (23:20 -0700)]
Create parquet datasets of reddit submissions from pushshift.

15 months agoRename spark script to reflect that it is for comments.
Nate E TeBlunthuis [Fri, 3 Jul 2020 21:00:36 +0000 (14:00 -0700)]
Rename spark script to reflect that it is for comments.

15 months agoupdate .gitignore
Nate E TeBlunthuis [Fri, 3 Jul 2020 20:55:25 +0000 (13:55 -0700)]
update .gitignore

15 months agobugfix in retrieving old data and rename file.
Nate E TeBlunthuis [Fri, 3 Jul 2020 20:54:55 +0000 (13:54 -0700)]
bugfix in retrieving old data and rename file.

15 months agoScript for checking shas for submissions.
Nate E TeBlunthuis [Fri, 3 Jul 2020 20:35:46 +0000 (13:35 -0700)]
Script for checking shas for submissions.

15 months agoBugfix: use timestamp types
Nate E TeBlunthuis [Fri, 3 Jul 2020 18:38:43 +0000 (11:38 -0700)]
Bugfix: use timestamp types

Also change the canonical file path.

15 months agoupdate the reddit comment dumps
Nate E TeBlunthuis [Fri, 3 Jul 2020 17:41:13 +0000 (10:41 -0700)]
update the reddit comment dumps

15 months agoDon't clobber old dumps so that we can just download the new ones.
Nate E TeBlunthuis [Fri, 3 Jul 2020 17:40:43 +0000 (10:40 -0700)]
Don't clobber old dumps so that we can just download the new ones.

15 months agoscript for getting submissions dumps from pushshift.
Nate E TeBlunthuis [Fri, 3 Jul 2020 00:40:17 +0000 (17:40 -0700)]
script for getting submissions dumps from pushshift.

15 months agoExtract variables from pushshift comment to parquet
Nate E TeBlunthuis [Thu, 2 Jul 2020 21:06:36 +0000 (14:06 -0700)]
Extract variables from pushshift comment to parquet

A spark script

Community Data Science Collective || Want to submit a patch?