cdsc_reddit.git
4 days agoMerge branch 'master' of code:cdsc_reddit master
Nate E TeBlunthuis [Tue, 6 Apr 2021 06:21:35 +0000 (23:21 -0700)]
Merge branch 'master' of code:cdsc_reddit

4 days agoChanges for cosine similarities on klone.
Nate E TeBlunthuis [Tue, 6 Apr 2021 06:21:06 +0000 (23:21 -0700)]
Changes for cosine similarities on klone.

2 weeks agoadd code for pulling activity time series from parquet.
Nate E TeBlunthuis [Wed, 24 Mar 2021 23:08:57 +0000 (16:08 -0700)]
add code for pulling activity time series from parquet.

6 weeks agoadd included_subreddits parameter to cosine similarities.
Nate E TeBlunthuis [Tue, 23 Feb 2021 02:38:34 +0000 (18:38 -0800)]
add included_subreddits parameter to cosine similarities.

6 weeks agoChanges from hyak.
Nate E TeBlunthuis [Tue, 23 Feb 2021 00:03:48 +0000 (16:03 -0800)]
Changes from hyak.

2 months agofix bug in viz.
Nate E TeBlunthuis [Thu, 28 Jan 2021 04:26:15 +0000 (20:26 -0800)]
fix bug in viz.

2 months agoadd visualization for 10000 subreddits based on author-tf similarities.
Nate E TeBlunthuis [Thu, 28 Jan 2021 04:22:24 +0000 (20:22 -0800)]
add visualization for 10000 subreddits based on author-tf similarities.

2 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Thu, 28 Jan 2021 04:09:23 +0000 (20:09 -0800)]
Merge branch 'master' of code:cdsc_reddit

2 months agoadd cluster selection to visualization
Nathan TeBlunthuis [Thu, 28 Jan 2021 04:08:07 +0000 (20:08 -0800)]
add cluster selection to visualization

3 months agoremove nsfw subs from topN
Nate E TeBlunthuis [Tue, 29 Dec 2020 05:11:44 +0000 (21:11 -0800)]
remove nsfw subs from topN

3 months agoUpdating to support wang-style user overlaps.
Nate E TeBlunthuis [Fri, 25 Dec 2020 06:38:04 +0000 (22:38 -0800)]
Updating to support wang-style user overlaps.

3 months agoSome improvements to run affinity clustering on larger dataset and
Nate E TeBlunthuis [Sun, 13 Dec 2020 04:42:47 +0000 (20:42 -0800)]
Some improvements to run affinity clustering on larger dataset and
compute density.

4 months agoRefactor and reorganze.
Nate E TeBlunthuis [Wed, 9 Dec 2020 01:32:20 +0000 (17:32 -0800)]
Refactor and reorganze.

4 months agoAdd code for running tf-idf at the weekly level.
Nate E TeBlunthuis [Wed, 2 Dec 2020 06:54:48 +0000 (22:54 -0800)]
Add code for running tf-idf at the weekly level.

4 months agorefactor visualization code.
Nathan TeBlunthuis [Wed, 18 Nov 2020 00:46:49 +0000 (16:46 -0800)]
refactor visualization code.

4 months agoMerge remote-tracking branch 'refs/remotes/origin/master' into master synced/master
Nathan TeBlunthuis [Wed, 18 Nov 2020 00:33:14 +0000 (16:33 -0800)]
Merge remote-tracking branch 'refs/remotes/origin/master' into master

4 months agogit-annex in nathante@nate-x1:~/cdsc_reddit
Nathan TeBlunthuis [Wed, 18 Nov 2020 00:33:13 +0000 (16:33 -0800)]
git-annex in nathante@nate-x1:~/cdsc_reddit

4 months agogit-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit
Nate E TeBlunthuis [Wed, 18 Nov 2020 00:31:48 +0000 (16:31 -0800)]
git-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit

4 months agoUpdate code for clustering + tsne.
Nate E TeBlunthuis [Tue, 17 Nov 2020 23:59:20 +0000 (15:59 -0800)]
Update code for clustering + tsne.

4 months agoUpdate code for building simlarity matrices.
Nate E TeBlunthuis [Tue, 17 Nov 2020 20:52:48 +0000 (12:52 -0800)]
Update code for building simlarity matrices.

4 months agobugfix in completing tfidf similarity matrices.
Nate E TeBlunthuis [Thu, 12 Nov 2020 19:47:53 +0000 (11:47 -0800)]
bugfix in completing tfidf similarity matrices.

4 months agoincrease learning rate.
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:58:39 +0000 (16:58 -0800)]
increase learning rate.

4 months agoincrease iterations and perplectity and early_exaggeration
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:55:39 +0000 (16:55 -0800)]
increase iterations and perplectity and early_exaggeration

4 months agoincrease learning rate
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:48:41 +0000 (16:48 -0800)]
increase learning rate

4 months agoFix bug in tsne.
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:43:41 +0000 (16:43 -0800)]
Fix bug in tsne.

4 months agogit-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:39:44 +0000 (16:39 -0800)]
git-annex in nathante@mox2.hyak.local:/gscratch/comdata/users/nathante/cdsc-reddit

4 months agosplit fitting and plotting tsne.
Nate E TeBlunthuis [Thu, 12 Nov 2020 00:38:22 +0000 (16:38 -0800)]
split fitting and plotting tsne.

4 months agoAdd file to plot related subreddits using tsne.
Nathan TeBlunthuis [Thu, 12 Nov 2020 00:05:36 +0000 (16:05 -0800)]
Add file to plot related subreddits using tsne.

4 months agoBugfix (typo)
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:38:11 +0000 (13:38 -0800)]
Bugfix (typo)

4 months agoReuse code for term and author cosine similarity.
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:18:57 +0000 (13:18 -0800)]
Reuse code for term and author cosine similarity.

4 months agoRefactor tfidf code to for code resuse.
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:18:19 +0000 (13:18 -0800)]
Refactor tfidf code to for code resuse.

4 months agorename 'idf' files to 'tfidf'
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:16:55 +0000 (13:16 -0800)]
rename 'idf' files to 'tfidf'

4 months agoImprovements to idf code
Nate E TeBlunthuis [Tue, 10 Nov 2020 21:12:11 +0000 (13:12 -0800)]
Improvements to idf code

5 months agoMerge branch 'master' of code:cdsc_reddit
Nate E TeBlunthuis [Mon, 2 Nov 2020 18:40:12 +0000 (10:40 -0800)]
Merge branch 'master' of code:cdsc_reddit

5 months agoadd term_cosine_similarity.py
Nate E TeBlunthuis [Mon, 2 Nov 2020 18:40:02 +0000 (10:40 -0800)]
add term_cosine_similarity.py

5 months agoAdd Cosine similarities to README.md
Nathan TeBlunthuis [Mon, 2 Nov 2020 17:48:10 +0000 (09:48 -0800)]
Add Cosine similarities to README.md

5 months agoUpdate Readme.
Nathan TeBlunthuis [Mon, 2 Nov 2020 16:42:13 +0000 (08:42 -0800)]
Update Readme.

5 months agoMerge branch 'master' of code:cdsc_reddit into master
Nathan TeBlunthuis [Mon, 2 Nov 2020 05:50:44 +0000 (21:50 -0800)]
Merge branch 'master' of code:cdsc_reddit into master

5 months agoCreate README.md
Nathan TeBlunthuis [Mon, 2 Nov 2020 05:50:27 +0000 (21:50 -0800)]
Create README.md

6 months agoUpdate reddit comments data with daily dumps.
Nate E TeBlunthuis [Sat, 3 Oct 2020 23:42:22 +0000 (16:42 -0700)]
Update reddit comments data with daily dumps.

7 months agoCompute IDF for terms and authors.
Nate E TeBlunthuis [Sun, 23 Aug 2020 18:57:55 +0000 (11:57 -0700)]
Compute IDF for terms and authors.

7 months agoUpdate submissions to parse using the backfill queue.
Nate E TeBlunthuis [Wed, 12 Aug 2020 05:37:36 +0000 (22:37 -0700)]
Update submissions to parse using the backfill queue.

7 months agobugfix in checking submission shas
Nate E TeBlunthuis [Tue, 11 Aug 2020 21:21:54 +0000 (14:21 -0700)]
bugfix in checking submission shas

7 months agoUse multiword expressions in tf.
Nate E TeBlunthuis [Mon, 10 Aug 2020 23:57:46 +0000 (16:57 -0700)]
Use multiword expressions in tf.

8 months agoFinish generating multiword expressions.
Nate E TeBlunthuis [Mon, 10 Aug 2020 05:42:23 +0000 (22:42 -0700)]
Finish generating multiword expressions.

8 months agoBugfix
Nate E TeBlunthuis [Sun, 9 Aug 2020 09:34:42 +0000 (02:34 -0700)]
Bugfix

8 months agoUse groupby - joins instead of windows
Nate E TeBlunthuis [Sun, 9 Aug 2020 07:21:50 +0000 (00:21 -0700)]
Use groupby - joins instead of windows

8 months agorenamte tf_comments part 2.
Nate E TeBlunthuis [Tue, 4 Aug 2020 20:39:49 +0000 (13:39 -0700)]
renamte tf_comments part 2.

8 months agorename tf_reddit_comments.py step1.
Nate E TeBlunthuis [Tue, 4 Aug 2020 20:39:20 +0000 (13:39 -0700)]
rename tf_reddit_comments.py step1.

8 months agoImprove tokenization following data. Generate author counts.
Nate E TeBlunthuis [Tue, 4 Aug 2020 20:24:37 +0000 (13:24 -0700)]
Improve tokenization following data. Generate author counts.

8 months agoimprove tokenizer.
Nate E TeBlunthuis [Tue, 4 Aug 2020 05:55:10 +0000 (22:55 -0700)]
improve tokenizer.

8 months agoTF reddit comments.
Nate E TeBlunthuis [Tue, 4 Aug 2020 05:43:57 +0000 (22:43 -0700)]
TF reddit comments.

8 months agocode to sort tf
Nate E TeBlunthuis [Tue, 4 Aug 2020 00:56:36 +0000 (17:56 -0700)]
code to sort tf

9 months agoremove is_submitter field from submissions which doesn't exist.
Nate E TeBlunthuis [Fri, 10 Jul 2020 00:12:14 +0000 (17:12 -0700)]
remove is_submitter field from submissions which doesn't exist.

9 months agoBugfixes in scripts.
Nate E TeBlunthuis [Wed, 8 Jul 2020 06:29:36 +0000 (23:29 -0700)]
Bugfixes in scripts.

9 months agoclean up comments in streaming example.
Nate E TeBlunthuis [Tue, 7 Jul 2020 19:28:57 +0000 (12:28 -0700)]
clean up comments in streaming example.

9 months agoupdate .gitignore
Nate E TeBlunthuis [Tue, 7 Jul 2020 19:28:44 +0000 (12:28 -0700)]
update .gitignore

9 months agoupdate examples with working streaming
Nate E TeBlunthuis [Tue, 7 Jul 2020 18:47:17 +0000 (11:47 -0700)]
update examples with working streaming

9 months agoBuild comments dataset similarly to submissions and improve partitioning scheme
Nate E TeBlunthuis [Tue, 7 Jul 2020 18:45:43 +0000 (11:45 -0700)]
Build comments dataset similarly to submissions and improve partitioning scheme

9 months agoupdate .gitignore
Nate E TeBlunthuis [Tue, 7 Jul 2020 07:58:26 +0000 (00:58 -0700)]
update .gitignore

9 months agoScript for example of streaming pyarrow.
Nate E TeBlunthuis [Tue, 7 Jul 2020 07:57:05 +0000 (00:57 -0700)]
Script for example of streaming pyarrow.

9 months agoScript to demonstrate reading parquet.
Nate E TeBlunthuis [Tue, 7 Jul 2020 07:51:40 +0000 (00:51 -0700)]
Script to demonstrate reading parquet.

9 months agoCheck the shas when we download dumps
Nate E TeBlunthuis [Tue, 7 Jul 2020 06:31:52 +0000 (23:31 -0700)]
Check the shas when we download dumps

9 months agoScript to run both parts of submissions_2_parquet.sh
Nate E TeBlunthuis [Tue, 7 Jul 2020 06:27:18 +0000 (23:27 -0700)]
Script to run both parts of submissions_2_parquet.sh

9 months agoCache before sorting so we don't extract twice.
Nate E TeBlunthuis [Tue, 7 Jul 2020 05:30:04 +0000 (22:30 -0700)]
Cache before sorting so we don't extract twice.

9 months agoMove the spark part of submissions_2_parquet to a separate script.
Nate E TeBlunthuis [Tue, 7 Jul 2020 05:26:29 +0000 (22:26 -0700)]
Move the spark part of submissions_2_parquet to a separate script.

9 months agoFix whitespace at top of file.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:32:00 +0000 (23:32 -0700)]
Fix whitespace at top of file.

9 months agoSecondary sort for the by_author dataset should be CreatedAt.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:27:18 +0000 (23:27 -0700)]
Secondary sort for the by_author dataset should be CreatedAt.

9 months agoCreate a second dataset sorted by author.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:24:40 +0000 (23:24 -0700)]
Create a second dataset sorted by author.

9 months agoCreate parquet datasets of reddit submissions from pushshift.
Nate E TeBlunthuis [Mon, 6 Jul 2020 06:20:17 +0000 (23:20 -0700)]
Create parquet datasets of reddit submissions from pushshift.

9 months agoRename spark script to reflect that it is for comments.
Nate E TeBlunthuis [Fri, 3 Jul 2020 21:00:36 +0000 (14:00 -0700)]
Rename spark script to reflect that it is for comments.

9 months agoupdate .gitignore
Nate E TeBlunthuis [Fri, 3 Jul 2020 20:55:25 +0000 (13:55 -0700)]
update .gitignore

9 months agobugfix in retrieving old data and rename file.
Nate E TeBlunthuis [Fri, 3 Jul 2020 20:54:55 +0000 (13:54 -0700)]
bugfix in retrieving old data and rename file.

9 months agoScript for checking shas for submissions.
Nate E TeBlunthuis [Fri, 3 Jul 2020 20:35:46 +0000 (13:35 -0700)]
Script for checking shas for submissions.

9 months agoBugfix: use timestamp types
Nate E TeBlunthuis [Fri, 3 Jul 2020 18:38:43 +0000 (11:38 -0700)]
Bugfix: use timestamp types

Also change the canonical file path.

9 months agoupdate the reddit comment dumps
Nate E TeBlunthuis [Fri, 3 Jul 2020 17:41:13 +0000 (10:41 -0700)]
update the reddit comment dumps

9 months agoDon't clobber old dumps so that we can just download the new ones.
Nate E TeBlunthuis [Fri, 3 Jul 2020 17:40:43 +0000 (10:40 -0700)]
Don't clobber old dumps so that we can just download the new ones.

9 months agoscript for getting submissions dumps from pushshift.
Nate E TeBlunthuis [Fri, 3 Jul 2020 00:40:17 +0000 (17:40 -0700)]
script for getting submissions dumps from pushshift.

9 months agoExtract variables from pushshift comment to parquet
Nate E TeBlunthuis [Thu, 2 Jul 2020 21:06:36 +0000 (14:06 -0700)]
Extract variables from pushshift comment to parquet

A spark script

Community Data Science Collective || Want to submit a patch?