Hee-Jun Jung

Hee-Jun Jung

Hee-Jun Jung is a Ph.D. candidate in AI at GIST, advised by Professor Kangil Kim. His research spans disentanglement learning, visual reasoning, and group theory.

Posts by Collection

portfolio

publications

Feature structure distillation with Centered Kernel Alignment in BERT transferring

Published in Expert Systems with Applications, 2023

How to transfer the feature structure from teacher to student model?

BibTeX: @article{JUNG2023120980, title = {Feature structure distillation with Centered Kernel Alignment in BERT transferring}, journal = {Expert Systems with Applications}, volume = {234}, pages = {120980}, year = {2023}, issn = {0957-4174}, doi = {https://doi.org/10.1016/j.eswa.2023.120980}, url = {https://www.sciencedirect.com/science/article/pii/S0957417423014823}, author = {Hee-Jun Jung and Doyeon Kim and Seung-Hoon Na and Kangil Kim}, keywords = {Knowledge distillation, BERT, Centered Kernel Alignment, Natural language processing}, abstract = {Knowledge distillation is an approach to transfer information on representations from a teacher to a student by reducing their difference. A challenge of this approach is to reduce the flexibility of the student’s representations inducing inaccurate learning of the teacher’s knowledge. To resolve the problems, we propose a novel method feature structure distillation that elaborates information on structures of features into three types for transferring, and implements them based on Centered Kernel Analysis. In particular, the global local-inter structure is proposed to transfer the structure beyond the mini-batch. In detail, the method first divides the feature information into three structures: intra-feature, local inter-feature, and global inter-feature structures to subdivide the structure and transfer the diversity of the structure. Then, we adopt CKA which shows a more accurate similarity metric compared to other metrics between two different models or representations on different spaces. In particular, a memory-augmented transfer method with clustering is implemented for the global structures. The methods are empirically analyzed on the nine tasks for language understanding of the GLUE dataset with Bidirectional Encoder Representations from Transformers (BERT), which is a representative neural language model. In the results, the proposed methods effectively transfer the three types of structures and improves performance compared to state-of-the-art distillation methods: (i.e.) ours achieve 66.61% accuracy compared to the baseline (65.55%) in the RTE dataset. Indeed, the code for the methods is available at https://github.com/maroo-sky/FSD.} }
Paper | Code

CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational AutoEncoder

Published in TMLR, 2024

How to represent symmetries for disentanglement learning without ground truth?

BibTeX: @article{ jung2024cfasl, title={{CFASL}: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational AutoEncoder}, author={Hee-Jun Jung and Jaehyoung Jeong and Kangil Kim}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2024}, url={https://openreview.net/forum?id=mDGvrH7lju}, note={} }
Paper | Code

talks

teaching

Teaching Assistant - Natural Language Processing (2022)

Graduate Course, GIST, AIGS, 2022

This cource gives comprehensive lectures and projects about NLP. (All materials have been kept strictly confidential.)

Chapter 0. Preparation

03/03 Lecture 1. Course Introduction

03/03 Lecture 2. Programming Language Basics for Project

Chapter 1. Background of AI

03/08 Lecture 3. Overview of AI

03/10-03/15 Lecture 4. Problem Solving

03/17 Lecture 5. Model Representation

03/22-03/31 Lecture 6. Optimization & Generalization

04/05 Lecture 7. Evaluation

04/07-04/14 Lecture 8. Probabilistic Modeling

Basics

Bayesian Network

Markov Random Field

04/19-04/21 Lecture 9. Logic Modeling

Background

Predicate Calculus

Inductive Logit Programming

Mid-term exam -> Replaced to Lab 2 Project

Chapter 2. Statistical Relational Learning

04/26-04/28 Lecture 10. Overview of SRL

Hyperplane-based Modeling - Neural Networks

Why SRL?

05/03-05/12 Lecture 11. Integration of Logic and Probability

Bayesian Logic Programming

Markov Logic Network

Logic Modeling: Basics

Logic Modeling: Formal Grammar

Probabilistic Context Free Grammar

Chapter 3. Natural Language Processing

05/17-05/26 Lecture 12. Overview of Natural Language Processing

Linguistics Essentials + More Terms

NLP Problems and Their Relations

N-gram

05/31 Lecture 13. Language Model

06/02 Lecture 14. Machine Translation

06/07 Lecture 15. Question Answering

06/09 Lecture 16. Conversational System

06/14 Final-Term Exam

  • Date: June 14, 2022
  • Time: 10:30 a.m ~ 11:45 a.m.
  • Venue: EECS C2 B101
  • Exam: Whole lectures. No coding assignment contents.

Projects

Project contents introduce basic practical knowledge different to the contents of the lectures. To cover the simple knowledge and programming issues, we will have 6 lab times in this semester (in our class time, no additional lab times are assigned.)

  • Lab 0, 1, 2, 3 : will be uploaded every 2 weeks
  • Lab 4, 5 : will be uploaded every 3 weeks

CHECK the key date of each assignment: (To be uploaded; Q&A Session; Deadline)

  • To be uploaded: the slide and video about the introduction of Lab N will be upload on time (not real-time). You can start from this date.
  • Q&A Session: interactive Q&A Session will be held on time (on Zoom meeting; real-time).
  • Deadline: you should submit your assignment before the deadline

Lab 0 - Environmental Settings (slide, video) [0%]

  • Schedule: (03.10; 03.15; No deadline)
  • Links: (kaggle, github) (In this assignment, no links)
  • Materials:
    • How to Use Colab (Machine Setting (Google Colab)/ guide for getting started)
    • How to Use GIST SW Center GPU Machine (user manual kor / eng) (Official video and document from SW Center) (GPU Server Specifications: 1 GPU; 3 CPU; 12GB RAM; 100GB HDD
    • How to submit your code and results (Kaggle / Github Repository)

Lab 1 - Representation of Symbolic Data (no slide, video) [10%]

  • Schedule: (03.24; 03.29; 04.06 23:59 KST)
  • Links: (kaggle pb1, pb2 / github)
  • Materials: Please see the materials on Github classroom

Lab 2 - NLP data preparation (slide, video) [10%; mid-term project]

  • Schedule: (04.07; 04.12; 04.20 23:59 KST / 04.20 14:59 UTC)
  • Links: (kaggle, github)
  • Materials: Please see the contents on kaggle and the slide

Lab 3 - Problem Formulation (slide, video) [10%]

  • Schedule: (04.21; 04.26; 05.04 23:59 KST / 05.04 14:59 UTC)
  • Links: (kaggle pb1, pb2 / github)
  • Materials: Please see the contents on Kaggle and the Slide

Lab 4 - Encoder-Decoder Implementation (slide, video - part1, 2) [10%]

  • Schedule: (05.06; 05.12; 05.25 23:59 KST / 05.25 14:59 UTC)
  • Links: (kaggle, github)
  • Materials: Please see the contents on kaggle and the slide

Lab 5 - Transformer Implementation (slide, video) [10%]

  • Schedule: (05.26; 05.31; 06.15 23:59 KST / 06.15 14:59 UTC)
  • Links: (kaggle, github)
  • Materials: Please see the contents on Kaggle and the Silde