#使用Transformer进行文本分类#代码提交 by YinHang2515 · Pull Request #937 · PaddlePaddle/book

YinHang2515 · 2020-11-30T07:47:33Z

项目地址：https://aistudio.baidu.com/aistudio/projectdetail/1247954

Text Classification with Transformer

CLAassistant · 2020-11-30T07:48:05Z

All committers have signed the CLA.

chenxiaozeng · 2020-12-07T12:14:52Z

+   "source": [
+    "import paddle\n",
+    "import paddle.nn as nn\n",
+    "import paddle.fluid.dygraph as dg\n",


Paddle2.0不建议使用fluid，默认动态图开发模式。

chenxiaozeng · 2020-12-07T12:16:23Z

+    "pad_id = word_dict['<pad>']\r\n",
+    "embed_dim = 32  # Embedding size for each token\r\n",
+    "num_heads = 2  # Number of attention heads\r\n",
+    "ff_dim = 32  # Hidden layer size in feed forward network inside transformer\r\n",


ff_dim变量命名不是很清晰。

chenxiaozeng · 2020-12-07T12:17:51Z

+    "        x = self.drop2(x)\r\n",
+    "        x = self.soft(x)\r\n",
+    "        return x\r\n",
+    "# class MyNet(paddle.nn.Layer):\r\n",


此处注释可删除。

chenxiaozeng · 2020-12-07T12:20:04Z

+   },
+   "source": [
+    "可以看到经过两轮的迭代训练，可以达到85%左右的准确率，当然你也可以通过调整参数、更改优化方式等等来进一步提升性能。"
+   ]


可使用model.predict进行预测，打印出句子，预测标签和实际标签，这样比较直观。

根据要求进行了相应的修改，并已同步更新至AIStudio

YinHang2515 · 2020-12-09T12:00:08Z

根据要求进行了相应的修改，并已同步更新至AIStudio

chenxiaozeng

2 suggestions.

chenxiaozeng · 2020-12-11T09:41:54Z

+    "class PointWiseFeedForwardNetwork(nn.Layer):\r\n",
+    "    def __init__(self, embed_dim, feed_dim):\r\n",
+    "        super(PointWiseFeedForwardNetwork, self).__init__()\r\n",
+    "        self.linear1 = pd.fluid.dygraph.Linear(embed_dim, feed_dim, act='relu')\r\n",


多处fluid需要改成nn

chenxiaozeng · 2020-12-22T10:04:02Z

+    "              loss=nn.CrossEntropyLoss())\r\n",
+    "\r\n",
+    "# 模型训练\r\n",
+    "model.fit(train_loader,\r\n",


训练完成之后，可以调用model.predict()测试下模型在test数据集上的表现。

chenxiaozeng · 2020-12-28T09:43:32Z

@@ -0,0 +1 @@
+


this file, to delete?

YinHang2515 · 2021-01-04T17:06:15Z

根据要求进行了相应的修改，并已同步更新至AIStudio

chenxiaozeng

LGTM

guoshengCS · 2021-01-13T03:05:41Z

+   },
+   "outputs": [],
+   "source": [
+    "class TransformerBlock(nn.Layer):\r\n",


Paddle中已经提供了Transformer的相关API https://www.paddlepaddle.org.cn/documentation/docs/zh/2.0-rc1/api/paddle/nn/layer/transformer/TransformerEncoder_cn.html#transformerencoder ，如果只是为了使用而不是要说明这些具体实现的话，可否直接使用这些API呢

TCChenlong

除了上述问题外，还有两处需要注意下：
1、2.0已经发布了，麻烦更新到2.0版本；
2、看预测的效果不是特别好，可以再优化一下网络
感谢~

TCChenlong · 2021-03-03T03:51:41Z

+   },
+   "outputs": [],
+   "source": [
+    "import paddle as pd\n",


import paddle

TCChenlong · 2021-03-03T03:52:55Z

+   "source": [
+    "import paddle as pd\n",
+    "import paddle.nn as nn\n",
+    "import paddle.nn.functional as func\n",


暂时不推荐这么写

TCChenlong · 2021-03-03T03:53:44Z

+    "train_dataset = IMDBDataset(train_sents, train_labels)\r\n",
+    "test_dataset = IMDBDataset(test_sents, test_labels)\r\n",
+    "\r\n",
+    "train_loader = pd.io.DataLoader(train_dataset, places=pd.CPUPlace(), return_list=True,\r\n",


places=pd.CPUPlace() 可以删除

YinHang2515 added 6 commits November 29, 2020 23:23

Text Classification with Transformer

812c36d

Merge pull request #1 from YinHang2515/YinHang2515-patch-1

cfec87d

Text Classification with Transformer

Delete text_classification_with_tansformer.ipynb

bd26e5e

Create 123

2db0f67

Add files via upload

3579b34

Delete 123

777ce82

chenxiaozeng reviewed Dec 7, 2020

View reviewed changes

YinHang2515 added 9 commits December 9, 2020 12:50

Add files via upload

6652be1

根据要求进行了相应的修改，并已同步更新至AIStudio

Delete text_classification_with_tansformer.ipynb

8143ba0

Delete text_classification_with_transformer.ipynb

1562032

Create text_classification_with_transformer

e08af53

Delete text_classification_with_transformer

e9919dd

Create 123

6078858

Add files via upload

d18f65b

Update text_classification_with_transformer.ipynb

7c71bcd

Update text_classification_with_transformer.ipynb

74cfdbc

chenxiaozeng reviewed Dec 22, 2020

View reviewed changes

chenxiaozeng reviewed Dec 28, 2020

View reviewed changes

Comment thread paddle2.0_docs/text_classification_with_transformer/123 Outdated

@@ -0,0 +1 @@

Copy link
Copy Markdown

chenxiaozeng Dec 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file, to delete?

YinHang2515 added 2 commits January 4, 2021 18:04

Add files via upload

9ff1814

Delete 123

f7cbe0c

chenxiaozeng approved these changes Jan 12, 2021

View reviewed changes

guoshengCS reviewed Jan 13, 2021

View reviewed changes

YinHang2515 added 2 commits January 14, 2021 22:54

已按照要求改用paddle自带api

f070a06

已按照要求调用相应api

c8937ba

TCChenlong reviewed Mar 3, 2021

View reviewed changes

Conversation

YinHang2515 commented Nov 30, 2020

Uh oh!

CLAassistant commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YinHang2515 commented Dec 9, 2020

Uh oh!

chenxiaozeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YinHang2515 commented Jan 4, 2021

Uh oh!

chenxiaozeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TCChenlong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CLAassistant commented Nov 30, 2020 •

edited

Loading